The Sqale method: presentation

Source Code Quality Evaluation: The SQALE method SQALE: Software Quality Assessment based on Lifecycle ExpectationsDecember 2011Author: Jean-Louis Letouzey

Agenda The needs for a Source Code evaluation method Issues with current aggregation rules The SQALE method structure The SQALE Quality Model The SQALE Analysis Model The SQALE indices and indicators SQALE in practice 2

The SQALE Method: Summary Software Quality Assessment based on Lifecycle Expectations Has been developed by experts, independent of any tool vendor Focus on the diagnostic objectivity (precision, no false-positive) Easy to understand, to implement and to deploy Avoids practices that damage measurement results Aggregation with averages that generates compensation effects Notation on a delimited interval that generates threshold effect Promotes simple principles Source code quality is a non functional requirement that should be specified, then verified Evaluating quality is measuring the remaining needed workload to fix all non-conformities Is a robust method for identifying and managing the Technical Debt

inspearit and source code evaluation inspearit is not a tool vendor, inspearit is an independant company inspearit thinks that the method come first, then the tools to support it inspearit assist large accounts to implement source code analysis with SQALE Our customers reported us issues with current methods and tools Difficulties to understand the meaning and usage of indicators Too much false positive No support for remediation decision (what are the priorities?) 5

The needs for a source codeevaluation method Ability to objectively evaluate and monitor software development products in order to anticipate issues Aligned with best measurement practices Ability to compare Source code versions Different products with different usage/history Development teams or subcontractors performance Capacity to provide useful inputs to an improvement plan Capacity to support decisions: Ex. Two teams working on two similar projects The first one, delivered 3 weeks in advance but with 100 coding practice issues The second, delivered 1 week in advance but with 15 coding practice issues Which project is the most efficient and effective? 6

High level requirements for anevaluation method Quantified, Objective, Precise, Sensitive Implementable by automated static analysis tools Reproducible by the implementation of one tool to another (produce the same findings based on non ambigous definitions, rules…) Provide guidance for tailoring this standard model to: Any language Different severity levels (business critical, life critical etc.)The SQALE method has been developped as a solution to all these requirements 7

Summary of the challenge foran evaluation method Two hierarchies: Challenge / Need: AH QH Provide a quality related “measure” or “score” for each A Hierarchy A Hierarchy of Quality couple {A,Q} of the 2 hierarchies of Artifacts char. and sub-char. = Quality Model Portfolio Quality Q Domain Maintainability CharacteristicA AppliA Sub-characteristic Component Measure/rule File Get a measure, a score which characterizes, represents the evaluated concept 9

Measurement basics: The representation condition “The condition that, if one software entity is less than another entity in terms of a selected attribute, then any software measure for that attribute must associate a smaller number to the first entity than it does to the second entity” [1] Real World Mathematical World Measurement function Attribute Mes(Aa) Impact on : - Measure/rule choice Aa - Normalization functions Mes(Ab)Artifacts - Aggregation rules Ab Mes(Ac) A representative Ac measure keep the relationship established in the Mes(Aa) > Mes(Ab) Aa >> Ab >> Ac real world Mes(Ab) > Mes(Ac) [1] N.E. Fenton and S. L. Pfleeger, Software Metrics: A rigourous & Practical Approach, 10 second edition, ISBN 053495425-1,PWS Publishing Company, Boston, 1997

The representation conditionapplied to aggregation ? Aggregation The aggregation should represent rule the basic findings Issues should be reported up to the highest level of the hierarchyIssues: AgScore_a AgScore_b The aggregate score should report the improvement Version b >> Version a AgScore_b > AgScore_aVersion a Improvement Version b We have identified some effects that violate this condition 11

The masking effectThe masking effect appears when the aggregate value is notsensitive to the variation of one of the base valuesExample: n ia 1 2 3 4 5 ed ax le le le le le in Fi Fi Fi Fi Fi M M M MyAppli Va A A C E E A E C MyAppli Vb A A C D E A E C An improvement from Va to Vb on file 4 does not impact the aggregate score Aggregations by Min, Max and Median violate the representation condition 12

The compensation effect The compensation effect appears on aggregation functions such as: mean, weighted mean, median Example: Comment ratio (target for the project : >30% per file) ge 10 ra 1 2 3 4 5 6 7 8 9 ve le le le le le le le le le le Fi Fi Fi Fi Fi Fi Fi Fi Fi Fi AMyAppli Va 5% 34% 48% 47% 31% 37% 33% 35% 4% 39% 31% While 2 files do have “maintainability” issue, the average is OK In real life, lack of comment in files 1 & 9 won’t be compensated by abundance of comments in file 3 & 4 Aggregations by average (weighted or not) violate the representation condition That ‘s one reason why most Quality Dashboards are not precise 13

The type of scale andallowed aggregations The measurement theory is precise about allowed aggregation [1] Scale Valid Transformation Main Valid Agregation Nominal 1 to 1 mapping None Ordinal Monotonic increasing Min, Max, Median function More Choice Interval M = aM + B (a>0) Min, Max, Median, Average Ratio M = aM (a>0) Min, Max, Median, Mean, Average, Sum, Distance (Euclidian or other) Absolute M = M All Due to the representation condition, some combinations (scale type, aggregation) should be rejected [1] N.E. Fenton and S. L. Pfleeger, Software Metrics: A rigourous & Practical Approach, 14 second edition, ISBN 053495425-1,PWS Publishing Company, Boston, 1997

Synthesis of our analysisSynthesis of allowed operations and aggregation issues depending on scale type Average, Weighted Scale Min, Max, median Sum, Distance average Not allowed Not allowed Not allowed Nominal Potentially not Not allowed Not allowed Ordinal representative Potentially not Potentially not Not allowed Interval representative representative Potentially not Potentially not Representative Ratio representative representative Potentially not Potentially not Representative Absolute representative representative Within SQALE, we choose to normalize all measures on a ratio scale and to aggregate the normalized values by summation 15

The SQALE method: Structure Implementation/Tools Tailoring 4 concepts 8 Fundamental Principles Measurement theory and representativity 17

The SQALE 8 Fundamental Principles1. The quality of the source code is a non-functional requirement2. The requirements in relation to the quality of the source code have to be formalised according to the same quality criteria such as any other functional requirement3. Assessing the quality of a source code is in essence assessing the distance between its state and its expected quality objective4. The SQALE Method assesses the distance to the conformity with the requirements by considering the necessary remediation cost of bringing the source code to conformity5. The SQALE Method respects the representation condition6. The SQALE Method uses addition for aggregating the remediation costs and for calculating its quality indicators7. The SQALE Method’s Quality Model is orthogonal8. The SQALE Method’s Quality Model takes the software’s lifecycle into account 18

The 4 main concepts ofthe SQALE method Static analysis tools Source Code1 Quality Model 2 Analysis Model 3 Indices 4 Indicators Testabilité Fiabilité Evolutivité Efficacité Maintenabilité Maintenabilité 589 Efficacité 248 248 SQI Remediation Evolutivité 1 480 1 480 1 480 List of source functions Fiabilité 548 548 548 548 code related Findings Remediation Testabilité 6 535 6 535 6 535 6 535 6 535 Aggregation STI costs table 6 535 7 083 8 563 8 811 9 400 requirements Table rules SRI … SQID … 19

Back to the fundamentals of QualityYou buy a new car• How will you feel, if the delivered car has only 5 cylinders and 290 hp? 21

Back to the fundamentals of Quality In 1979, Philip Crosby in his famous book “Quality is free” established the 4 principles of Quality: the definition of quality is conformance to requirements the system of quality is prevention the performance standard is zero defects the measurement of quality is the price of nonconformance Since that time, the vision and definition of quality has been extended to a much wider scope including customer satisfaction But anyway, quality is still at least “Conformance to requirements” 22

The SQALE Quality Model: source code requirements An organized set of expectations (requirements) Requirement based on lifecycle needs Sub-characteristic Understandability Reuse Readability Reusability Ram related efficiency Portability Rom related efficiency Maintain Maintainability Characteristic CPU related efficiency Archi. related changeability Deliver Reusability Security … Logic related changeability Efficiency … Portability Data related changeability no public data Evolve Changeability Maintainability Fault tolerance Number of derived class <=10 Reliability … Security Architecture related reliability Test Logic related reliabilityTestability Efficiency Instruction related reliability Code Changeability Data related reliability Reliability Integration Testing testability Testability Unit Testing testability Requirements, appear only once within the Quality Model, when they are first needed. They are checked with relevant static analysis tools 23

Requirements: type of issues It is important to use a SQALE Quality Model that covers all the types of code issues Useless code, un-optimized codePresentation, structurness Excessive coupling, Maintainability Hard coded data Efficiency Changeability Potential logic errors, exception management, Reliability test coverage Testability Copy and Paste, internal structure of methods When deployed, the SQALE Quality Model contains from 30 to 100+ requirements tailored to the organization context 24

The SQALE Analysis Model Quality Indexes represent the remediation effort needed to refactor artifacts in order to comply with the Quality ModelPart level indexes are aggregated by adding all file indexes Part level indexes Σ Sub characteristic indexes Remediation 4.1 Analysis function Tool Non conformity table Remediation cost table Σ “Understandability index” for the selected file 4.1“Characteristic indexes” are aggregated by adding “Subcharacteristic indexes” 26

The SQALE Analysis Model For a given couple {A,Q}, SQALE provides a simple rule to Quality calculate the associated score Q Maintainability Characteristic Sub-characteristic Portfolio Measure/rule Domain Req_1 Req_2 Req_3 Req_4 Req_5 Req_6 … … … … Req_99 File_1 File_2 A AppliA File_3 File_4 … Σ … Component … … … File File_99999 Remediation costs tableThe positions into the 2 hierarchies define the perimeter of remediation costs to be added 27

The SQALE Analysis Model:remediation factorsHow findings are transformed into costs?• SQALE use « Remediation Functions » that are associated to types of Non Conformity. The standard SQALE Analysis Model contains 5 types which correspond to different « remediation lifecycles »• These Types and values are proposed by default. I it is recommended to extend /taylor them at Organization/Project/Application level Estimated cost for fixing one Non Conformity of Type4 is: 5 Work Units 28

3° concept: SQALE indicesThe SQALE characteristic indices: Artifact remediation workload - SQALE Testability Index: STI for all requirements - SQALE Reliability Index: SRI associated to testability - SQALE Changeability Index: SCI - SQALE Security Index: SSI - SQALE Efficiency Index: SEI - SQALE Maintainability Index: SMI - SQALE Portability Index: SPI - SQALE Reusability Index: SRuIThe global SQALE Quality Index: SQI - This is the Technical DebtSQALE index densities: SQID, STID etc Example: Density by KLOC (1,000 lines of code) 30

4° concept: The 3 SQALE indicatorsSQALE Indices SQI STI SQALE Kiviat SRI … SQID Rating Grid … Based on the ratio (in %) Testabilité Fiabilité Evolutivité Efficacité Maintenabilité Remediation cost / Development cost Maintenabilité 589 Efficacité 248 248 Example of Testability rating Evolutivité 1 480 1 480 1 480 Remediation cost (STI): 4.36 hours Fiabilité 548 548 548 548 Development cost: 250 hours Testabilité 6 535 6 535 6 535 6 535 6 535 Ratio: 1.7 % Rating: “C” 6 535 7 083 8 563 8 811 9 400 31

The SQALE Pyramid: A two points of view indicatorInterpreting the SQALE indices An external view that represents the percieved quality evaluated by consolidation of the hierarchy of characteristicsAn analytic view provided byorthogonal characteristicsOne understands impact of Maintainabilityeach Non Conformity and Σ Efficiencyimprovement on qualitycharacteristic and life cycle Changeability Σissues. Reliability Σ Σ Testability © Copyright inspearit 2007-20011 32

The perceived benefits As SQALE requires to specify the quality of the code, the quality measure is objective The SQALE quality index represents a workload, a cost. It is the concrete “Technical debt” of the project It is easy to monitor simultaneously: the remaining workload associated to functionalities the debt associated to code quality and update accordingly the project‘s planning Technical debt may be aggregated at any portfolio granularity Technical debt density allows to compare versions, applications, subcontractors…As SQALE does not violate the representation condition, SQALE thusprovides usable meanings to source code measurements 33

Using SQALE: Tools PRIVATE, METRIXWARE:SQALE index distribution analysis 35

Using SQALE: ToolsSQUORE: Dashboard 36

Using SQALE: ToolsSONAR: Monitoring the source code quality(Technical Debt) 37

Using SQALE: ToolsSONAR: Tailored Dashboard 38

Using SQALE: ToolsSONAR: Portfolio Management 39

Using SQALE: Deployment 0 Initialization 1 Tailoring 2 Implementation 3 Deployment Method training Planification Choose and Tool Deployment, Development of Tailored Stake holders Implementation of training coaching SQALE models Perimeter … the solution Monitor and improve Specify the tool solution Identify the On day training for Choose the solution to Perform awareness sponsor and stake the team: The SQALE be implemented within session holders, define the Method the organization roadmap Coach and support the Develop a tailored Implementation of the users Define the most SQALE Quality Model tailored models within usefull use cases the selected solution Monitor the solution and of source code Develop a tailored define an improvement analysis: SQALE Analysis Model Implementation of the plan (identification and selected indicators and implementation of new Build the « Source Validation of both reports within the requirements) code analysis » models trough a pilot selected solution Project team project Update process, Specify the tool associated deliverables solution including and training to prepare recommanded deployment indicatorsSupport Training, Support Seminars Workshop preparation Training and animation Coaching Support 40

The SQALE Discovery KitDiscover the fundamentals principles and benefits of the SQALE method with the“SQALE Discovery Kit”. This package contains: A one day training session on the SQALE Method The identification (through dedicated interviews) of your main “use cases” of source code analysis within your organisation’s context The development (through dedicated Workshops) of your own quality and analysis models These models will be tailored to your environment and will be the basis for defining and evaluating the quality of your source code (for one of the following language: Java, C, C++, Cobol) The concrete assessment of one of your application using the SQALE method and your tailored quality and analysis models including a detailed assessment report Workshop on how to interpret and use the results At the end you will get: Your tailored models for one of your development language An evaluation report Direction for actions 41 Total duration: about 20 days

ThanksTo know more about SQALE: http: /www.sqale.orgTo do more with SQALE, contact inspearit just sqale it © inspearit - Author - Place 42 Saturday, December 17, 2011

The Sqale method: presentation

by Jean-Louis LETOUZEY on Dec 13, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 16

Statistics

The Sqale method: presentation — Presentation Transcript

http://www.linkedin.com	14
http://paper.li	2