Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Techniques to Speed Up Your Build Pipeline 
ashish.parkhi@ideas.comnaresh@agilefaqs.com 
@AshishParkhi@nashjain 
ashishpar...
Build Pipeline -Best Case to Worst Case time. 
About60 to90 minutes
Impact on life 
Image source – http://ak3.picdn.net/shutterstock/videos/5132438/preview/stock-footage-mixed-ethnicity-grou...
Build Pipeline –Now takes 10 to 12 Minutes
Focus on the Bottlenecks 
Divide and Conquer 
Key Principles to Speed Up Your Build Pipeline 
Fail Fast
Commercial Break 
ashish.parkhi@ideas.comnaresh@agilefaqs.com 
@AshishParkhi@nashjain 
ashishparkhi.comnareshjain.com
Focus on the Bottlenecks 
Divide and Conquer 
Key Principles to Speed Up Your Build Pipeline 
Fail Fast
Disk IO –Example 
File Operations 
Focus on Bottleneck
Disk IO –Example 
Database operations. Image Source - https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTPdVawndjUZbU2...
Disk IO –Alternative 
Image Source - http://3.bp.blogspot.com/-bqTjSN7pSpg/UbqyjVojEFI/AAAAAAAADBw/PWe0kiuRHJ4/s200/no+dup...
Disk IO –Alternative 
•Avoid file operations –e.g. Jar creation. 
Focus on Bottleneck 
Image Source - http://i1.wp.com/blo...
Disk IO –Alternative 
Image source - http://4.bp.blogspot.com/_4hvqisoH9CE/TSZIs7eiSAI/AAAAAAAAA7E/vanj6bGD8XQ/s1600/big-v...
Disk IO –Alternative -SSD 
CrystalDiskMark - http://crystalmark.info/software/CrystalDiskMark/index-e.html 
•HDD (Toshiba ...
Disk IO –Alternative -SSD 
•HDDSSD 
Focus on Bottleneck
Disk IO –Alternative -In Memory DB 
Memory (Heap) Engine 
–had some limitations over myisamengine. 
Focus on Bottleneck
Disk IO –Alternative -In Memory DB 
–was not supporting many MySQL queries so was discarded. 
Focus on Bottleneck
Disk IO –Alternative -In Memory DB 
database 
–looked promising as it could support many MySQL queries but still required ...
Disk IO –Alternative -In Memory DB 
–looked most promising as it is wire compatible with MySQL, which means without code c...
Disk IO –Alternative -RAM Drive 
•SoftPerfect RAM Disk 
Focus on Bottleneck
Disk IO –Alternative –RAM Drive 
•RAM Drive 
Focus on Bottleneck
Disk IO –Alternative –RAM Drive 
•RAM Drive –did not work 
Focus on Bottleneck
Disk IO –Alternative –RAM Drive 
•RAM Drive –did not work 
Focus on Bottleneck
CPU -Profiling 
Focus on Bottleneck
CPU –Profiling -Insights 
Image source - https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcQde6NeSrbuv40CIhKtFa1OuIQXf...
CPU –Profiling -Insights 
Image source - http://2.bp.blogspot.com/-uKMyLlB3F7o/Tqn_6yqdElI/AAAAAAAAB94/_1FMbHJFQBQ/s1600/w...
CPU –Profiling -Insights 
Image source - http://dilbert.com/dyn/str_strip/000000000/00000000/0000000/000000/00000/7000/000...
CPU –Profiling -Insights 
java.util.Calendaris horribly slow. 
Total processing time took 20.72minutes out of which Date A...
CPU –Profiling -Insights 
java.util.Calendaris horribly slow. We switched to jodadate library and deprecated java.util.Dat...
CPU -ANT 1.7 Junit task options. 
Focus on Bottleneck
Focus on the Bottlenecks 
Divide and Conquer 
Key Principles to Speed Up Your Build Pipeline 
Fail Fast
CPU -Running Tests Concurrently 
•Create parallel jobs. 
Divide and Conquer
CPU -Running Tests Concurrently 
•Distribute tasks across multiple slaves. 
Divide and Conquer 
Image source -https://wiki...
CPU -Running Tests Concurrently 
Image source - http://sharpreflections.com/wp-content/uploads/2012/06/multi_core_cpu.png ...
Focus on the Bottlenecks 
Divide and Conquer 
Key Principles to Speed Up Your Build Pipeline 
Fail Fast
Restructure The Build Pipeline 
Image Source - http://javapapers.com/wp-content/uploads/2012/11/failfast.jpg 
•We want our...
Incremental Build vs. Clean Build 
•Local dev builds are incremental, instead of clean builds, as it helps with faster fee...
Prioritize Test 
•We prioritize and group our tests so that the tests which are fast and mostly likely to fail are run fir...
Summary 
•Focus on bottlenecks 
–Avoid Disk IO -File operations, file based database operations. 
–Use smaller datasets. 
...
Build Time Vs No Of Builds 
Removed Workspace Duplication 
Ant Junit Task –Fork Once 
Ram Disk 
Caching Resource 
Caching ...
Impact on life 
Image source - http://t3.gstatic.com/images?q=tbn:ANd9GcTCvK8pY5qcp7Gl3ZBjxN1mc1HVHdiy1sQhByKeGgUk_5eJuUk7...
Resources 
•Jenkins –http://jenkins-ci.org/ 
•CI –http://en.wikipedia.org/wiki/Continuous_integration 
•Mklink –http://tec...
© Copyright Integrated Decisions and Systems, Inc. (IDeaS –A SAS COMPANY) 
Visit IDeaS online at www.ideas.com 
Thank you....
Techniques to Speed up your Build Pipeline
Upcoming SlideShare
Loading in …5
×

Techniques to Speed up your Build Pipeline

2,356 views

Published on

I would like to share my experience and journey on how we brought down our Jenkins build pipeline time down from over 90 minutes to under 12 minutes. In the process, I would share specific techniques which helped and also some, which logically made sense, but actually did not help. If your team is trying to optimize their build times, then this session might give you some ideas on how to approach the problem.

Development Impact - For one of our build job, below graph shows how the number of builds in a day have increased over a period of time as the build time has reduced. Frequency of code check-in has increased; Wait time has reduced; failed test case faster to isolate and fix.

Details: http://confengine.com/agile-pune-2014/proposal/458/techniques-to-speed-up-your-build-pipeline-for-faster-feedback

Conference: http://pune.agileindia.org/

Published in: Software
  • Be the first to comment

Techniques to Speed up your Build Pipeline

  1. 1. Techniques to Speed Up Your Build Pipeline ashish.parkhi@ideas.comnaresh@agilefaqs.com @AshishParkhi@nashjain ashishparkhi.comnareshjain.com
  2. 2. Build Pipeline -Best Case to Worst Case time. About60 to90 minutes
  3. 3. Impact on life Image source – http://ak3.picdn.net/shutterstock/videos/5132438/preview/stock-footage-mixed-ethnicity-group-of-medical-professionals-working-late-at-night-are-looking- at-a-computer.jpg http://the247analyst.files.wordpress.com/2011/10/dealing-with-pressure.jpg http://www.dimitri.co.uk/business/business-images/worker-alone-dark-office.jpg http://cdn.sheknows.com/articles/2012/10/crying-little-girl.jpg
  4. 4. Build Pipeline –Now takes 10 to 12 Minutes
  5. 5. Focus on the Bottlenecks Divide and Conquer Key Principles to Speed Up Your Build Pipeline Fail Fast
  6. 6. Commercial Break ashish.parkhi@ideas.comnaresh@agilefaqs.com @AshishParkhi@nashjain ashishparkhi.comnareshjain.com
  7. 7. Focus on the Bottlenecks Divide and Conquer Key Principles to Speed Up Your Build Pipeline Fail Fast
  8. 8. Disk IO –Example File Operations Focus on Bottleneck
  9. 9. Disk IO –Example Database operations. Image Source - https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTPdVawndjUZbU2PDn-oKgjBPqmgDqr3PPZatZh9kxEgNi71AND http://www.dba-oracle.com/images/large_disk_hot_files.gif Focus on Bottleneck
  10. 10. Disk IO –Alternative Image Source - http://3.bp.blogspot.com/-bqTjSN7pSpg/UbqyjVojEFI/AAAAAAAADBw/PWe0kiuRHJ4/s200/no+duplicate+content.jpg •Avoid file operations –e.g. duplicating workspace Focus on Bottleneck
  11. 11. Disk IO –Alternative •Avoid file operations –e.g. Jar creation. Focus on Bottleneck Image Source - http://i1.wp.com/blog.quoteroller.com/wp-content/uploads/2013/04/Dont-start-from-scratch.png?resize=800%2C264
  12. 12. Disk IO –Alternative Image source - http://4.bp.blogspot.com/_4hvqisoH9CE/TSZIs7eiSAI/AAAAAAAAA7E/vanj6bGD8XQ/s1600/big-vs-small-left.jpg •Test on smaller but apt data set. Focus on Bottleneck
  13. 13. Disk IO –Alternative -SSD CrystalDiskMark - http://crystalmark.info/software/CrystalDiskMark/index-e.html •HDD (Toshiba MQ01ACF050 500GB SATA III) vs SSD (Samsung PM851 512GB mSata) Focus on Bottleneck
  14. 14. Disk IO –Alternative -SSD •HDDSSD Focus on Bottleneck
  15. 15. Disk IO –Alternative -In Memory DB Memory (Heap) Engine –had some limitations over myisamengine. Focus on Bottleneck
  16. 16. Disk IO –Alternative -In Memory DB –was not supporting many MySQL queries so was discarded. Focus on Bottleneck
  17. 17. Disk IO –Alternative -In Memory DB database –looked promising as it could support many MySQL queries but still required couple of modifications to our code. Focus on Bottleneck
  18. 18. Disk IO –Alternative -In Memory DB –looked most promising as it is wire compatible with MySQL, which means without code changes I could just point to memsql and be done with it. Focus on Bottleneck
  19. 19. Disk IO –Alternative -RAM Drive •SoftPerfect RAM Disk Focus on Bottleneck
  20. 20. Disk IO –Alternative –RAM Drive •RAM Drive Focus on Bottleneck
  21. 21. Disk IO –Alternative –RAM Drive •RAM Drive –did not work Focus on Bottleneck
  22. 22. Disk IO –Alternative –RAM Drive •RAM Drive –did not work Focus on Bottleneck
  23. 23. CPU -Profiling Focus on Bottleneck
  24. 24. CPU –Profiling -Insights Image source - https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcQde6NeSrbuv40CIhKtFa1OuIQXf7F7esMJKp1Ie7zmH2t29l6Z Scanningresource bundle files from jars. Focus on Bottleneck
  25. 25. CPU –Profiling -Insights Image source - http://2.bp.blogspot.com/-uKMyLlB3F7o/Tqn_6yqdElI/AAAAAAAAB94/_1FMbHJFQBQ/s1600/weight-lift-cartoon.jpg Loading Spring Application Context. Focus on Bottleneck
  26. 26. CPU –Profiling -Insights Image source - http://dilbert.com/dyn/str_strip/000000000/00000000/0000000/000000/00000/7000/000/7029/7029.strip.gif Avoiding unnecessary activities during build e.g. sending out email. Focus on Bottleneck
  27. 27. CPU –Profiling -Insights java.util.Calendaris horribly slow. Total processing time took 20.72minutes out of which Date Arithmetic took 18.15minutes which is about 87.6%of the total processing time! Focus on Bottleneck
  28. 28. CPU –Profiling -Insights java.util.Calendaris horribly slow. We switched to jodadate library and deprecated java.util.DateAPI. Now Date Arithmetic takes 1.30minutes; that’s a massive saving of 93.77% Focus on Bottleneck
  29. 29. CPU -ANT 1.7 Junit task options. Focus on Bottleneck
  30. 30. Focus on the Bottlenecks Divide and Conquer Key Principles to Speed Up Your Build Pipeline Fail Fast
  31. 31. CPU -Running Tests Concurrently •Create parallel jobs. Divide and Conquer
  32. 32. CPU -Running Tests Concurrently •Distribute tasks across multiple slaves. Divide and Conquer Image source -https://wiki.jenkins-ci.org/download/attachments/2916393/logo.png?version=1&modificationDate=1302753947000
  33. 33. CPU -Running Tests Concurrently Image source - http://sharpreflections.com/wp-content/uploads/2012/06/multi_core_cpu.png •Using @RunWith(ConcurrentJunitRunner.class). –Curtesy -Mathieu Carbou http://java.dzone.com/articles/concurrent- junit-tests –Maven Surefire plugin has built in mechanism. Divide and Conquer
  34. 34. Focus on the Bottlenecks Divide and Conquer Key Principles to Speed Up Your Build Pipeline Fail Fast
  35. 35. Restructure The Build Pipeline Image Source - http://javapapers.com/wp-content/uploads/2012/11/failfast.jpg •We want our builds to give us fast feedback. Hence it is very important to prioritise your build tasks based on what is most likely to fail first. •Push unnecessary stuff to a separate build –Things like JavaDocs can be done nightly. •Separate out fast and slow running tests. Fail Fast
  36. 36. Incremental Build vs. Clean Build •Local dev builds are incremental, instead of clean builds, as it helps with faster feedback and fail fast. Fail Fast
  37. 37. Prioritize Test •We prioritize and group our tests so that the tests which are fast and mostly likely to fail are run first. –ProTest framework Fail Fast
  38. 38. Summary •Focus on bottlenecks –Avoid Disk IO -File operations, file based database operations. –Use smaller datasets. –Use in-memory databases, Ram Drives, SSDs. –Perform CPU profiling, scan logs, to uncover the unknown. –Verify build tool settings. •Divide and Conquer –Create smaller jobs that can run in parallel. –Distribute jobs across multiple slaves. –Write tests that can run in isolation and use ConcurrentJunitRunnerto run them in parallel. •Fail Fast –Restructure the build pipeline to uncover failures soon. –Incremental Builds –Prioritize tests.
  39. 39. Build Time Vs No Of Builds Removed Workspace Duplication Ant Junit Task –Fork Once Ram Disk Caching Resource Caching Spring Context Avoided Email Joda DateTime Deprecated Date API Concurrent Junit Runner
  40. 40. Impact on life Image source - http://t3.gstatic.com/images?q=tbn:ANd9GcTCvK8pY5qcp7Gl3ZBjxN1mc1HVHdiy1sQhByKeGgUk_5eJuUk7cA https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcQpoUXqhEpdGl1cLzn4gQsng_GyxUmOKWxYUH6GfrjN_FRUYPxw-Q
  41. 41. Resources •Jenkins –http://jenkins-ci.org/ •CI –http://en.wikipedia.org/wiki/Continuous_integration •Mklink –http://technet.microsoft.com/en-us/library/cc753194.aspx •http://ant.apache.org/manual/Tasks/junit.html •http://java.dzone.com/articles/javalangoutofmemory-permgen •SSD –http://en.wikipedia.org/wiki/Solid-state_drive •Hybrid disk –http://en.wikipedia.org/wiki/Hybrid_drive •HSQL –http://hsqldb.org/ •H2 –http://www.h2database.com/html/main.html •Memsql –http://www.memsql.com/ •MySQL is bazillion times faster than MemSQL •Tmpfs –http://en.wikipedia.org/wiki/Tmpfs •http://blog.laptopmag.com/faster-than-an-ssd-how-to-turn-extra-memory-into-a-ram-disk •RAM Disk Software Benchmarked •http://jvmmonitor.org/ •http://searchvmware.techtarget.com/tip/VMware-snapshot-size-and-other-causes-for-slow- snapshots •http://blogs.agilefaqs.com/2014/10/03/key-principles-for-reducing-continuous-integration-build- time/ •http://googletesting.blogspot.com/2011/06/testing-at-speed-and-scale-of-google.html •http://www.infoq.com/presentations/Development-at-Google •http://crystalmark.info/software/CrystalDiskMark/index-e.html
  42. 42. © Copyright Integrated Decisions and Systems, Inc. (IDeaS –A SAS COMPANY) Visit IDeaS online at www.ideas.com Thank you. ashish.parkhi@ideas.comnaresh@agilefaqs.com @AshishParkhi@nashjain ashishparkhi.comnareshjain.com

×