Improving Quality through Continuous Integration - A case study of CollabNet

1,120 views
911 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,120
On SlideShare
0
From Embeds
0
Number of Embeds
54
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Improving Quality through Continuous Integration - A case study of CollabNet

  1. 1. 1 Copyright ©2014 CollabNet, Inc. All Rights Reserved.ENTERPRISE CLOUD DEVELOPMENT Improving Quality through Continuous Integration A case study of CollabNet best practices Janardhanam Venkat Director Engineering Dr. Dobbs Conference - Pune - April 13th 2014
  2. 2. 2 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Overview • Introduction to Basic CI • Chaos in Product Development • Defining CI strategy • Building a CI strategy • Execution cycle feedback • CI Infrastructure • Drill down to Metrics • Measurement & Maturity
  3. 3. 3 Copyright ©2014 CollabNet, Inc. All Rights Reserved. CI Packaging CI Server Ping Version Control CI Server Commit - Detailed Test Suite Longer Running Test Light weight Test Basic CI Environment - Unit Test - Check Style
  4. 4. 4 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Chaos & Problem Where to start?
  5. 5. 5 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • Not sure how much unit test coverage is written for a feature • Not sure if my code is following best practices • No time to test in multiple browsers and their versions • Unable to cope up with installer testing – Combination of OS & database has many permutations to test – Turn around time to test a minor installer fix is 2 weeks or longer Agile Team Problem
  6. 6. 6 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • Infrastructure Security Vulnerability – Prior Product release versions in production • Application Security Vulnerability – Cross site scripting, SQL injection • Service Issues – SOAP API breaking • Broken Link in the product – Help contents 404 • Few feature have issue that are browser specific – Not working in IE 9 Customer Problem Reactive (Customer Problem) Proactive (Agile Teams Problem)
  7. 7. 7 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • Use provisioning to enable CI • Run parallel tests • Tie code review process with commit tests • Test in clone of Production environment Industry Best Practices
  8. 8. 8 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Input to CI Strategy CI Strategy Customer Problem Agile Team Problem Industry Best Practices Roadmap
  9. 9. 9 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • Parameters that defines the strategy – What kind of test Coverage? • Certain features may need more coverage – How important is Time to Feedback? • Can team act upon if the feedback takes longer – What is the Cost? • Hardware versus Tester Resource • Pick license or Open Source tool Refining CI strategy Source: Revello Systems, March 2014 You may not get all three parameters right
  10. 10. 10 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Building CI strategy Problem Tools Value Infrastructure Security Vulnerability Nessus Identify JDK, Apache, OS vulnerability Application Security Testing Acunetix XSS, SQL injection Installer Automation Home grown Multi platform testing Functional Testing Selenium Reduce time Services Testing SOAP UI Automate Static Code Analysis FindBugs, PMD Code Quality Broken Links Link Checker Identify 404 Code Coverage Cobertura Unit test coverage Performance Testing JMeter Load Testing Browser Compatibility Testing Browsera Multi browser test 508 Compliance Testing WAT
  11. 11. 11 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Extending CI Infrastructure Self Provisioning ALM Jenkins CI Infrastructure
  12. 12. 12 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Leverage Self Provisioning Environment • Provisioning of boxes – Platform, OS & application configuration – E.g.: Solaris, CentOS, Suse, RHEL, in combination with 32/64 Bit, 2/4/8 core & different storage space – Common Profile for teams Datacenter Distributed Teams SDLC Mgmt. Customer (Early release) Product – Ops 1 Product – QA 1 Product – QA 2 Product – QA 3 Product – Dev 1
  13. 13. 13 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Leverage ALM for Traceability
  14. 14. 14 Copyright ©2014 CollabNet, Inc. All Rights Reserved. CI Master Nessus Suite Suite 1 Suite 2 … Suite 6 CI Master Detail Test Suite Suite 1 Suite 2 …. Suite 10 Commit Test CI Master Functional Test CI Master SOAP Broken Link CI Master Installer Automation Case 1 Case 2 …… Case 14 FindBug PMD CI Infrastructure – Self Provisioning, ALM & Jenkins Boxes Provisioned in private cloud Master Slave Around 70 Servers ALM
  15. 15. 15 Copyright ©2014 CollabNet, Inc. All Rights Reserved. CI Execution Cycle – Feedback Commit Validation SOAP Services Test Static Code Analysis Functional Test Installer Automation Application Security Testing Infra. Security 4 Hr 1 Week10 Min 20 Min 3 Hr 10 Hr1 Hr Detail Unit Test Commit reverted for failed test Defect created for failed test 15 Min0
  16. 16. 16 Copyright ©2014 CollabNet, Inc. All Rights Reserved. CI Execution Cycle – Feedback Commit Validation SOAP Services Test Static Code Analysis Functional Test Installer Automation Application Security Testing Infra. Security 4 Hr 1 Week10 Min 20 Min 3 Hr 10 Hr1 Hr Detail Unit Test Commit reverted for failed test Defect created for failed test 15 Min0 598 1327 5601 5287 93 8 0 1000 2000 3000 4000 5000 6000 Commit Validation SOAP Selenium Full Unit Test Load Test Installer Automation #ofTestCases TeamForge Automated Cases
  17. 17. 17 Copyright ©2014 CollabNet, Inc. All Rights Reserved. CI Visibility Dashboard Took an year to implement the strategy Started in April 2013
  18. 18. 18 Copyright ©2014 CollabNet, Inc. All Rights Reserved. CI Visibility Notification
  19. 19. 19 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Metrics - Customer Reported Core Product Defects 17.20 15.13 7.00 0.00 2.00 4.00 6.00 8.00 10.00 12.00 14.00 16.00 18.00 20.00 Jun-12 Dec-12 Jun-13 Dec-13 TeamForge Release Date Customer Defect By Release Ratio
  20. 20. 20 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Metrics - Installer Automation • Reduce Installer qualification time to 1 day – Phase I – Total 185 hours to qualify one round of installer testing – After automating 8 installer cases, reduced qualification time to 134 hours Manual, 131 hrs Saved, 51 hrs Actual, 3 hrs. Automated, 54 Installer Automation - 8 of 19 cases done One round manual qualification - Total 185 hours
  21. 21. 21 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • CI increases confidence and reduces risk • Defines quality, prevention & reduction of defect • Builds value with faster feedback • Shorten software release time • Results published, increases visibility to agile teams • Challenges Continues – Unable to automate data warehouse ETL testing – Keeping up with tools & maintenance overhead – Large scale refactoring & architecture changes CI Value to Agile Teams
  22. 22. 22 Copyright ©2014 CollabNet, Inc. All Rights Reserved. • Infrastructure vulnerability - Setup in prior 3 releases • Broken link - Set up in prior product release • Setting up the same CI infrastructure for 3 branches Additional Refinements
  23. 23. 23 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Measure & Mature CI Strategy Basic Find defect after development Manual Deployment Team actions on report Commit Validation Manual Build Intermediate Early detection Partial auto deploy to Test Historical & analysis Code analysis, Functional Test Automated Build & CI Advanced Reduce customer defect Auto deploy to Test /Stage Trending Reports Performance, Security Test Leverage provisioning Optimized Defect prevention Auto deploy to Prod Predictive Modeling Services Test Track CI config changesBuild Test Visibility Deployment Quality
  24. 24. 24 Copyright ©2014 CollabNet, Inc. All Rights Reserved.24 Copyright ©2014 CollabNet, Inc. All Rights Reserved. Questions? Janardhanam Venkat [venkatj@collab.net] www.collab.net +1-650-228-2500 +1-888-778-9793 blogs.collab.net twitter.com/collabnet www.facebook.com/collabnet www.linkedin.com/company/collabnet-inc

×