• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Process Improvement in High Maturity Environments   Improving the Efficiency of Code Reviews
 

Process Improvement in High Maturity Environments Improving the Efficiency of Code Reviews

on

  • 218 views

 

Statistics

Views

Total Views
218
Views on SlideShare
218
Embed Views
0

Actions

Likes
0
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Process Improvement in High Maturity Environments   Improving the Efficiency of Code Reviews Process Improvement in High Maturity Environments Improving the Efficiency of Code Reviews Presentation Transcript

    • Portugal Process Improvement in High Maturity Environments Improving the Efficiency of Code Reviews André Ferreira Software Quality Assurance Engineer Critical Software S.A. 2013-10-18
    • Outline © 2013 CMMI Portugal Conference Series – All Rights reserved. 1. SPI success - evaluation and measurement 2. CMMI High Maturity Processes Areas - a framework to drive SPI initiatives 3. SPI initiative at Critical Software – Improving the efficiency of Code Reviews 4. Recommendations/guidelines to improve the probability of success of SPI initiatives. 2
    • SPI success: reports on SPI Success. Medium Improvement # Data points Lowest Improvement Highest Improvement Cost © 2013 CMMI Portugal Conference Series – All Rights reserved. Performance category 34% 29 3% 87% Schedule 50% 22 2% 95% Productivity 61% 20 11% 329% Quality 48% 34 2% 132% Customer Satisfaction 14% 7 -4% 55% Return on Investment [4.0:1] 22 [1.7:1] [27.7:1] Performance Results of CMMI-Based Process Improvement –Technical Report CMU/SEI-2006-TR-004 ESC-TR-2006-004 3
    • SPI success: evaluation of SPI Success - I © 2013 CMMI Portugal Conference Series – All Rights reserved. Evaluation and validity of SPI success is questionable. Hard to establish a clear relation of SPI initiative and SPI results. Results are not questionable by themselves, but a clear understanding of how success relates to SPI is unclear, mainly as result of: 1. 2. 3. 4. 5. Confounding factors – not considering relevant factors, validity threat Incomplete context descriptions – context of SPI not fully described Evaluation validity – threats to validity not discussed Measurement validity – measurement definition inconsistencies Measurement scope – missing the big picture Unterkalmsteiner et.al. (2011). Evaluation and Measurement of Software Process Improvement A Systematic Literature Review. IEEE Transactions on Software Engineering 4
    • © 2013 CMMI Portugal Conference Series – All Rights reserved. SPI success: evaluation of SPI Success - I Causation established Presence of confounding factors Y(SPI sucess) = f ( f1, f2, f,3) Y’(SPI sucess) = f’ (?, ?, f1, f2, f3, ?) 5
    • Process Management Project Management © 2013 CMMI Portugal Conference Series – All Rights reserved. CMMI High Maturity Framework Overview Support BO OPM PPBs Project QPPOs OPP QPM OIP QPPOs CAR Defects Performance Issues Corrective Actions 6
    • © 2013 CMMI Portugal Conference Series – All Rights reserved. SPI Initiative Improving the Efficiency of Code Reviews 7
    • SPI Initiative – Opportunity Identification OPM.SP 1.1/OPM.SP1.2 © 2013 CMMI Portugal Conference Series – All Rights reserved. 1. Annual Review Business Objectives   Business objective: Improve organizational profit margins QPPOs: Improve project profit margins – do the same/better for less 2. Feedback from projects  Issues with review procedure – cumbersome - to much time managing review records.  Lead to establish a performance baseline of total appraisal cost for code reviews. 8
    • SPI Initiative – Opportunity Identification © 2013 CMMI Portugal Conference Series – All Rights reserved. OPM.SP1.2/OPM.SP 1.3  Analysis of performance data  Feedback from projects was confirmed to be an issue – statistical analysis of a control chart.  Code review is a - fat process!  Opportunity identified:  Decrease appraisal cost of performing code review procedure by:  Improving efficiency (less effort) but control impact on effectiveness 9
    • SPI Initiative – Potential Improvement I OPM.SP 1.3 𝐶𝑜𝑄 = 𝑠𝑢𝑚(𝐼 + 𝐸 + 𝐴 + 𝑃) High Cost © 2013 CMMI Portugal Conference Series – All Rights reserved. Cost of Quality (CoQ) Management model Total cost of quality (I) Internal and (E) External failure costs (A) Appraisal and (P) Preventive costs Low 100% defective Quality 100% good 10
    • SPI Initiative – Potential Improvement I OPM.SP 1.3 Process Management © 2013 CMMI Portugal Conference Series – All Rights reserved. Rationale of impact on QPPOs BO Project Gross Margin >= Target QPPOs Processes Decrease on overall CoQ will have a positive impact in the ability of projects to meet Project Gross Margin (QPPO) EBT Verificati on/ Code reviews 𝐶𝑜𝑄 = 𝑠𝑢𝑚(𝐼 + 𝐸 + 𝐴 + 𝑃) Rework 11
    • SPI Initiative – Potential Improvement II OPM.SP 1.3 High Cost © 2013 CMMI Portugal Conference Series – All Rights reserved. Hypothesis1: Less effort spent performing code reviews will contribute to a reduction in total appraisal cost. Total cost of quality Internal and external failure costs Appraisal and preventive costs Low 100% defective Quality 100% good 12
    • SPI Initiative – Potential Improvement II OPM.SP 1.3 © 2013 CMMI Portugal Conference Series – All Rights reserved. Hypothesis1: Less effort spent performing each code review will contribute to a reduction in total appraisal cost  Rationale of estimation of potential benefit  (1) Assuming a 10x decrease on managing review records -> x hours saved per review  (2) y number of reviews performed each year Total benefit (TH1)  TH1(hours) = x*y  TH1(€) = ~4K€/year 13
    • SPI Initiative – Potential Improvement III OPM.SP 1.3 © 2013 CMMI Portugal Conference Series – All Rights reserved. Hypothesis 2: Less effort per code review can be used to perform additional code reviews (re-invest in Appraisal Cost)   Motivate an increase of # code reviews in projects with little investment in code reviews Motivate re-investment of effort saved to perform additional code reviews (increase code coverage) ...with the final objective of decrease Total Cost of Quality by decreasing considerably the Internal and External Failure costs. 14
    • SPI Initiative – Potential Improvement III OPM.SP 1.3 High Cost © 2013 CMMI Portugal Conference Series – All Rights reserved. Hypothesis 2: Less effort per code review can be used to perform additional code reviews (Increase Appraisal Costs) Total cost of quality Internal and external failure costs Appraisal and preventive costs Low 100% defective Quality 100% good 15
    • SPI Initiative – Potential Improvement III OPM.SP 1.3  Estimation of potential benefit © 2013 CMMI Portugal Conference Series – All Rights reserved.    Based on performance baselines it takes 5x more to fix a defect at ST when compared to code reviews Projects who invest in code reviews, on average, have less defects/LOC found in ST (corroborates the model) with a typical distribution of [74%/24%] distribution of code reviews VS. ST defects Projects who invest less in code reviews have [5%/94%] defect distribution In the scenario of these projects with lower investment in code review shifting to a bigger investment in code reviews, having an approximation to a ratio of [74%/24/] we estimated potential savings of 70K€ for a year timeframe. TH2(€) = 70K€ 16
    • SPI Initiative – ROI OPM.SP 1.3 © 2013 CMMI Portugal Conference Series – All Rights reserved. 𝑟𝑒𝑡𝑢𝑟𝑛 − 𝑖𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡 𝑖𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡  Estimation of potential benefit ROI =  Hypothesis 1  Tool license cost: 3.6K  Management of SPI initiative: Deployment + Training: 3K€  ROI [ 0.4:1]  Hypothesis 2  Tool license cost: 3.6K  Management of SPI initiative: Deployment + Training: 3K€  ROI [ 9.6:1] 17
    • SPI Initiative – Elicit Improvements © 2013 CMMI Portugal Conference Series – All Rights reserved. OPM.SP2.1/SP2.2  A target was given to identify approaches that could decrease in 10x the time to manage code review records  Several tools were identified and analyzed in a DAR initiative – ‘smoke test’.  Crucible came as the best option as it enabled to set-up a review in less than 10 minutes among other relevant factors related to infrastructure. 18
    • SPI Initiative – Pilot © 2013 CMMI Portugal Conference Series – All Rights reserved. OPM.SP2.3/SP2.4  Criteria of success in supporting H1.  Decrease review setup/manage records is consistent below 10m/code review  Impact on defect detection ability (def/kloc) is non negative  Pilot lasted 3 months and involved 5 project and a workshop with SPA area members. 19
    • SPI Initiative – Pilot Results OPM.SP2.4 © 2013 CMMI Portugal Conference Series – All Rights reserved.  Performance baseline with pilot results was created.  Review setup/manage records was under 10m resulting in a 12x decrease on overall effort – change in performance tested with statistical significance.  Defect defection rate has not affected: when compared to existing performance baseline (44 defect/KLOC) Overall, Crucible supported H1 and therefore a base to approach H2 with a higher degree of confidence. 20
    • SPI Initiative – Deployment OPM.SP3.1/SP3.2/SP3.3 © 2013 CMMI Portugal Conference Series – All Rights reserved.  Phase currently in progress – Evaluation of ROI will be over a 1 year period – Monitoring cost of quality – For the second year end beyond the NPV will replace ROI for Cost/Benefit analysis of SPI initiative 21
    • SPI Recommendations © 2013 CMMI Portugal Conference Series – All Rights reserved. Seek to establish your Governance Model (GM)    Use/create a GM that supports the rationale of your network of Objectives vs. Processes Start small, measure the least as possible, focus on making the model reliable. Only with a minimal statistical confidence on cause and effect you are able to estimate ROI of SPI Y(Repeatable SPI sucess) = f (f1,f2,f3) Y’(Risky SPI initiative) = f‘(?, ?, f1, f2, ?) 22
    • © 2013 CMMI Portugal Conference Series – All Rights reserved. You can contact me at: al-ferreira@criticalsoftware.com or csw-quality@criticalsoftware.com Questions? 23