Advertisement
Advertisement

More Related Content

Advertisement

Oh Crap, My Code is Slow - Madison PHP 2016

  1. Oh Crap, My Code is Slow Chris Tankersley @dragonmantank Madison PHP Conference, September-October 2016 Madison PHP Conference, September-October 2016 1
  2. Madison PHP Conference, September-October 2016 2 1.79 MHz 8-bit Processor 128K RAM 640x192 max resolution 64 color palette RS-232 Serial Port Cartridge Bay 2 Joystick Ports Disk Extended Color Basic 2.1
  3. Madison PHP Conference, September-October 2016 3 520 Mhz Apple S1 512MB RAM 390x312 resolution (~303 ppi density) 16 million colors WatchOS
  4. Madison PHP Conference, September-October 2016 4 32 Cores 512GB RAM 1-10Gbps NICs 10 Terabyte FusionIO Cards
  5. Madison PHP Conference, September-October 2016 5
  6. “premature optimization is the root of all evil.” Donald Knuth, “Structured Programming With Go To Statements” Madison PHP Conference, September-October 2016 6
  7. Madison PHP Conference, September-October 2016 7 www.phpbench.com
  8. It Doesn’t Matter Madison PHP Conference, September-October 2016 8
  9. We Are the 3% Madison PHP Conference, September-October 2016 9
  10. Who is at fault? • 3rd Party Connections • I/O Performance • Database Madison PHP Conference, September-October 2016 10
  11. Madison PHP Conference, September-October 2016 11
  12. Madison PHP Conference, September-October 2016 12
  13. Madison PHP Conference, September-October 2016 13
  14. 4 Hours • Cache Ad Campaign Data • Cache Analytics Data • Run Numbers • Sync Products Madison PHP Conference, September-October 2016 14
  15. 4 Hours • Cache Ad Campaign Data • Cache Analytics Data • Run Numbers • Sync Products Madison PHP Conference, September-October 2016 15
  16. The Problems Madison PHP Conference, September-October 2016 16
  17. Running Numbers was heavy • Primary server would spike under load • Secondary servers would get out of sync and go into “rollback” Madison PHP Conference, September-October 2016 17
  18. Madison PHP Conference, September-October 2016 18 0 500 1000 1500 2000 2500 3000 5:20am 5:25am 5:30am 5:35am 5:40am 5:45am Replication Lag Lag in Seconds
  19. Madison PHP Conference, September-October 2016 19 0 1000 2000 3000 4000 5000 6000 5:20am 5:25am 5:30am 5:35am 5:40am 5:45am Replication Lag Lag in ms
  20. Product Sync was Slow • Just took hours to run Madison PHP Conference, September-October 2016 20
  21. Madison PHP Conference, September-October 2016 21
  22. Product Sync Madison PHP Conference, September-October 2016 22
  23. Just Start Logging • Add DEBUG log messages with timestamps • Where is it slow? Madison PHP Conference, September-October 2016 23
  24. Seldaek/monolog use MonologLogger; use MonologHandlerStreamHandler; // create a log channel $log = new Logger(‘job_debug'); $log->pushHandler(new StreamHandler('path/to/your.log', Logger::DEBUG)); // add records to the log $log->debug(date(‘Y-m-d H:i:s’) . ‘ – Contacted API’); // Do our business logic $log->debug(date(‘Y-m-d H:i:s’) . ‘ – Finished with page’); Madison PHP Conference, September-October 2016 24
  25. Culprits Madison PHP Conference, September-October 2016 25
  26. What did we find? • All calls to Product API had to be full sets, couldn’t subset • Calls to Product API were slow, but not horrid • Generating and inserting the Products were slow due to business logic • Blocked Operations: • Getting next page from API • Processing products Madison PHP Conference, September-October 2016 26
  27. Our Workflow // Original Workflow Get Page X from API For Each Product: Extract Data from XML Transmogrify the Data into a Product Object Save Object to DB If No Next Page: Break Else: Page++ Continue Madison PHP Conference, September-October 2016 27
  28. Solution – Out of Band Processing // Original Workflow Get Page X from API For Each Product: Extract Data from XML Transmogrify the Data into a Product Object Save Object to DB If No Next Page: Break Else: Page++ Continue Madison PHP Conference, September-October 2016 28
  29. Solution – Out of Band Processing // Job 1 - Cache Product API Calls Get Page X…X+10 from API Cache XML to Database If No Next Page: Break Else: Page++ Continue Call Job 2 Respawn Job Madison PHP Conference, September-October 2016 29
  30. Solution – Out of Band Processing // Job 2 – Insert Products Get Page X…X+10 from DB For Each Product: Extract Data from XML Transmogrify the Data into a Product Object Save Object to DB If No Next Page: Break Else: Page++ Continue Madison PHP Conference, September-October 2016 30
  31. Run Totals Madison PHP Conference, September-October 2016 31
  32. Madison PHP Conference, September-October 2016 32
  33. Background • Was originally PHP • Turned into a MongoDB Script because it was too slow Madison PHP Conference, September-October 2016 33
  34. Madison PHP Conference, September-October 2016 34 0 500 1000 1500 2000 2500 3000 5:20am 5:25am 5:30am 5:35am 5:40am 5:45am Replication Lag Lag in ms
  35. Madison PHP Conference, September-October 2016 35
  36. What did we find? Madison PHP Conference, September-October 2016 36
  37. Check the Server Metrics Madison PHP Conference, September-October 2016 37 https://aws.amazon.com/blogs/aws/new-cloudwatch- metrics-for-amazon-ebs-volumes/
  38. Suspect Madison PHP Conference, September-October 2016 38
  39. Our Solution – Throw Hardware At It Madison PHP Conference, September-October 2016 39
  40. Our Solution – Throw Hardware At It • Increased IOPs on the SSD’s • Larger Instances on AWS Madison PHP Conference, September-October 2016 40
  41. Our Solution – Move out of MongoDB • Rewrite the script back into PHP • Run in our worker system Madison PHP Conference, September-October 2016 41
  42. The Result Madison PHP Conference, September-October 2016 42
  43. The New Bug Madison PHP Conference, September-October 2016 43 It took 5 hours to run
  44. Suspect Madison PHP Conference, September-October 2016 44
  45. Madison PHP Conference, September-October 2016 45
  46. Madison PHP Conference, September-October 2016 46
  47. What did we find? Madison PHP Conference, September-October 2016 47
  48. Code Profiling Madison PHP Conference, September-October 2016 48
  49. xhprof • Low-cost dynamic analysis for PHP • PHP Extension • Store results in a DB • Has a pretty good GUI • https://www.digitalocean.com/community/tutorials/how-to-set-up- xhprof-and-xhgui-for-profiling-php-applications-on-ubuntu-14-04 Madison PHP Conference, September-October 2016 49
  50. Pretty Graphs Madison PHP Conference, September-October 2016 50
  51. Useful Metrics Madison PHP Conference, September-October 2016 51
  52. What we find? • Hydrating objects was expensive • We were doing deep hydration, resulting in extra DB and hydration calls • We had authentication checking happening in a loop, due to bad logging code Madison PHP Conference, September-October 2016 52
  53. The result? Madison PHP Conference, September-October 2016 53 It brought it down to around 3.5 hours
  54. Valgrind • General programming tool for checking memory debugging, memory leaks, and code profiling • Supported by xdebug • KCacheGrind/QCacheGrind to view output Madison PHP Conference, September-October 2016 54
  55. Enable it in xdebug zend_extension=/usr/lib/php/20151012/xdebug.so xdebug.profiler_enable=1 xdebug.profiler_output_dir=/var/www/tests/xdebug Madison PHP Conference, September-October 2016 55
  56. Function Calls and Code Flow Madison PHP Conference, September-October 2016 56
  57. What we find? • We were looping a lot • We were looping big loops inside small loops • We were looping through a lot of the same data multiple times Madison PHP Conference, September-October 2016 57
  58. The Result – Reduce the Looping Madison PHP Conference, September-October 2016 58 Runtime was reduced to 30 minutes
  59. Tips for Slow Code • Use Monolog to add Debugging messages • Use xhprof to profile “live” code • Use xdebug and Valgrind to get deeper profiling Madison PHP Conference, September-October 2016 59
  60. Thank You! • https://github.com/dragonmantank • Author of “Docker for Developers” • https://leanpub.com/dockerfordevs • http://ctankersley.com • chris@ctankersley.com • @dragonmantank Madison PHP Conference, September-October 2016 60
  61. Credits • Slide 13 – Andrei.D40 – Stacks of Books, Flickr • Slide 34 – Upper Snake River Valley Historical Society – 3339 loggin, Flickr Madison PHP Conference, September-October 2016 61

Editor's Notes

  1. Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.
Advertisement