Software Product Measurement and Analysis in a Continuous Integration Environment


Published on

Presentation of a paper presented in the International Conference ITNG 2010, about a framework constructed for software internal quality measurement program with automatic metrics extraction, implemented at a Software Factory.

1 Comment
  • how can i save the presentation???? pls tell me
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • The "why?" is important mainly because it defines how data should be interpreted providing the basis for reusing measurement plans and procedures for future projects and activities [5]. Figure 3 shows in the GQ(I)M the levels that can be applied to products, processes and resources in a life cycle of a system.
  • Following the GQ(I)M suggestion from a specialized work using Goal-Driven Software Measurement, it was found that the use of GQ(I)M complements the planning steps of MA of a PA within the CMMi [8]. The GQ(I)M top-down approach was applied to plan the measurement and analysis described in this paper, as shown in Table 2. Within the scope of this paper some product metrics were selected that could be automatically extracted from source codes and unit tests, with fixed frequency, through a Continuous Integration environment
  • The metrics organization in a structured database has allowed different ways of metrics analysis. Some of them were connecting client software like MS Excel to metrics database, generating statistical graphics, or exploring PivotTables [3].
  • But features and structure of transactional databases are tuned to support operational tasks like maintaining operational data by using On-Line Transaction Processing – OLTP. This solution might not be suitable if there is a large load of data metrics being collected and stored from a lot of projects, as in a Software Factory. In this case, would be necessary a structured database for high-performance analysis like a Data Warehouse (DW). Here, an approach for quickly answering multidimensional user queries would be also important, which could be achieved by using On-Line Analytical Processing (OLAP).
  • Once software metrics were loaded into a multidimensional database like a DW, they could be analyzed by using OLAP with good performance. That happened mainly because in the ETL processing, fact tables of metric values were aggregated by each possible grouping with the dimension tables attributes, and then a Cube was generated. So, the user could quickly execute common OLAP operations over the Metrics Cube. Among some used operations were the Drill-Down/Up and Slice-And-Dice, allowing users to analyze information in a more or less detailed view, or to change/combine different dimensions. Figure 5 shows an example of an OLAP analysis over the software metrics Cube. The matrix shows in lines the date/time of automated CI builds executed over time, and in columns it shows OO class names. In this example, matrix cells represent the Cyclomatic Complexity and Nb Lines of Code (LOC) metrics. So it can be seen how these metrics evolve during time, for each class. In this case, it can be noticed also that the Cyclomatic Complexity is increasing with the insertion of LOC, as expected. The user can Drill Up to analyze metrics in a higher level, from Package and Week perspectives, for example, or can also Slice data, like just filtering classes from one specific assembly.
  • The GQ(I)M approach preaches the importance of Graphical Indicators that makes sense even for a non-specialist. Generally, Indicators are understandable in a higher level than metrics, and are calculated over one or more metrics. In this work, it was generated a set of Key Performance Indicators (KPI) [8] that helped answering the Questions listed in the Table 2. An example of these KPI for a specific project, considering the desired Goal for them, is presented in Figure 6. This kind of approach has allowed the Project Manager to analyze, considering a time frame, how maintainable, reusable, and testable was the developed source code.
  • It was also possible the generation of a Kiviat graphic to allow an integrated analysis of the metrics maximum (desired) value against the measured values. As shown in Table 3 and Figure 8, in the case study’s project, the maximum measure for Cyclomatic Complexity (CC) found in a class was 87, whilst the assumed maximum desired CC value for a class was 50. That depicts that there are still some room to go improving the source code and reducing its complexity.
  • Another kind of performed analysis was the generation of statistical graphics. Figure 7 presents how the Unit Test Coverage was reduced in the project over time. From that, it can be inferred that, initially some unit tests were developed, but as the LOC increased in the next days, few new unit tests were created to test the inserted new methods.
  • Software Product Measurement and Analysis in a Continuous Integration Environment

    1. 1. Authors : Gabriel de Souza P. Moreira Roberto Pepato Mellado Denis Ávila Montini Prof. Dr. Luiz Alberto Vieira Dias Prof. Dr. Adilson Marques da Cunha
    2. 2. <ul><li>Abstract </li></ul><ul><li>Product Software Metrics </li></ul><ul><li>Software Quality Engineering Standards for SW Products </li></ul><ul><li>Measurement and Analysis - CMMI </li></ul><ul><li>Goal-Driven Measurement Process </li></ul><ul><li>GQ(I)M </li></ul><ul><li>Selected Metrics </li></ul><ul><li>Continuous Integration </li></ul><ul><li>Automatic Metrics Extraction Process </li></ul><ul><li>Software Metrics DW Model </li></ul><ul><li>Metrics Analysis </li></ul><ul><li>Conclusions </li></ul>
    3. 3. <ul><li>The paper describes a framework for a software internal quality measurement program with automatic metrics extraction. This framework was successfully implemented in an Industrial Software Factory. That was possible through the implementation of a proposed Continuous Integration (CI) environment to periodically analyze source codes and extract metrics. These metrics were consolidated in a Data Warehouse, allowing On-line Analytical Processing (OLAP) and Key Performance Indicator (KPI) analysis with high-performance and user-friendly interface. The measurement program followed GQ(I)M paradigm for metrics selection to ensure that collected metrics are relevant from the Software Factory goals perspective. Finally, the Measurement and Analysis Process Area of the Capability Maturity Model integration - CMMi was used for measurement and analysis planning and implementation. </li></ul>
    4. 4. <ul><li>Measurements can be used by software engineers to help assessing the quality of technical work products and to assist in tactical decisions during a project [1]. </li></ul><ul><li>To accomplish real-time quality assessment, engineers must use technical measures to evaluate quality in objective rather than in subjective ways. [1] </li></ul>
    5. 5. <ul><li>First generation : ISO/IEC 9126 [9]. </li></ul><ul><li>Second generation : ISO/IEC 25000 (SQuaRE) [7]. </li></ul><ul><li>Both ISO standard families are based on the model described in the bellow premisses: </li></ul>Internal metrics can be used as indicators that allow forecasting the system behavior during tests and operation phases and to prevent problems during the development process [9].
    6. 6. <ul><li>Measurement and Analysis (MA) is a supporting Process Area (PA) at the Maturity Level 2 of CMMI. </li></ul><ul><li>The purpose of MA is to develop and sustain a measurement capability that is used to support management information needs. [2] </li></ul>
    7. 7. <ul><li>The Goal-Driven Measurement Planning Process [3] proposes 10 steps for measurement planning. </li></ul><ul><li>Identify your business goals. </li></ul><ul><li>Identify what you want to know or learn. </li></ul><ul><li>Identify your subgoals. </li></ul><ul><li>Identify the entities and attributes related to your subgoals. </li></ul><ul><li>Formalize your measurement goals. </li></ul><ul><li>Identify quantifiable questions and related indicators that you will use to help achieve your measurement goals. </li></ul><ul><li>Identify the data elements that you will collect to construct the indicators that help answer your questions. </li></ul><ul><li>Define measures to be used, and make these definitions operational. </li></ul><ul><li>Identify the actions that you will take to implement the measures. </li></ul><ul><li>Prepare a plan for implementing the measures. </li></ul>G Q I M
    8. 8. <ul><li>Goal-Question-(Indicator)-Measure [3] was based on GQM paradigm from Basili [5]. </li></ul><ul><li>GQM and its GQ(I)M extension are useful because they facilitate to identify not only precise required measures, but also the reasons why the data are being collected. </li></ul>
    9. 9. (*) .NET specific metrics (**) OO specific metrics
    10. 10. <ul><li>NbLinesOfCode – means the number of logical lines of code. </li></ul><ul><li>*NbILInstructions – means the number of instructions in Intermediate Language (IL) of .NET Platform. </li></ul><ul><li>NbMethods – means the number of methods (abstract, virtual or non-virtual, constructor, property or indexer) present in a type. </li></ul><ul><li>NbFields – means the number of fields (field, enum, read only field or constant) present in a type. </li></ul><ul><li>Code Source Cyclomatic Complexity (CC) – means the number of decisions that can be taken in a procedure. The CC for a type is defined as the sum of its methods CC. </li></ul><ul><li>*IL Cyclomatic Complexity (ILCC) – means the .NET Language independent cyclomatic complexity measure. It is computed from IL as 1 + the number of different offsets targeted by a jump/branch IL instruction. </li></ul>
    11. 11. <ul><li>TypeRank – TypeRank values are computed by applying the Google’s PageRank algorithm on the graph of type dependencies. A homothety of center 0.15 is applied to make it so that the average of TypeRank is 1. </li></ul><ul><li>Lack Of Cohesion Of Methods (LCOM) – Single Responsibility Principle states that a type should not have more than one reason to change. Such a class is said to be cohesive. A high LCOM value generally pinpoints a poorly cohesive class. There are several LCOM metrics. The LCOM takes its values in the range [0-1] </li></ul><ul><li>Lack of Cohesion Of Methods Henderson-Sellers (LCOM HS) – the LCOM of HS (HS stands for Henderson-Sellers) that takes its values in the range [0 to 2]. A LCOM HS value highest than 1 should be considered alarming. </li></ul><ul><li>**NbChildren – the number of children for a class is the number of sub-classes (whatever their positions in the sub branch of the inheritance tree). The number of children for an interface is the number of types that implement it. </li></ul><ul><li>**Depth of Inheritance Tree (DIT) – means the Depth of Inheritance Tree for a class or a structure as its number of base classes. </li></ul>
    12. 12. <ul><li>NbLinesOfComment – means the sum of the number of lines of comment that can be found in each of its partial definition. </li></ul><ul><li>PercentageComment – means this metric is calculated by using the following formula: PercentageComment = 100 * NbLinesOfComment / ( NbLinesOfComment + NbLinesOfCode ). </li></ul><ul><li>Afferent coupling at type level (Ca) – means the Afferent Coupling for a particular type is the number of types that depends directly on it. </li></ul><ul><li>Efferent coupling at type level (Ce) – means the Efferent Coupling for a particular type is the number of types it directly depends on. </li></ul><ul><li>Association Between Classes (ABC) – means the Association Between Classes metric for a particular class or structure is the number of members of others types it directly uses in the body of its methods </li></ul><ul><li>PercentageCoverage – means the percentage of code covered by unit tests. </li></ul>
    13. 13. <ul><li>The CI [4] is a software engineering practice where developers frequently integrate their work in a centralized repository. </li></ul><ul><li>As the work is integrated, an automated build is executed to discover integration errors as quick as possible. </li></ul><ul><li>Some new approaches can be added to this CI practice, including but not limited to: unit testing, code coverage, metrics extraction, static code analysis. </li></ul>
    14. 14. <ul><li>Cruise Control.Net for CI; </li></ul><ul><li>Nant for automating build tasks; </li></ul><ul><li>NUnit for unit testing; </li></ul><ul><li>PartCover for extracting test code coverage; </li></ul><ul><li>NDepend for extracting software metrics. </li></ul>
    15. 15. <ul><li>The implemented process for automatic metrics extraction in the case study is presented below. </li></ul>Source-Code Static Analysis ( NDepend in CI) Source Code XML Processing MDB MDB ETL MDB DW OLAP XML
    16. 16. DW Model implemented in SQL Server Analysis Services 2008
    17. 17. OLAP analysis over a Software Metrics Cube
    18. 18. Example of SW Metrics Key Performance Indicators (KPI)
    19. 19. DIT CC LCM LCM HS EC Measured 3 87 0.98 1.08 42 Maximum (Desired) 5 50 1.00 2.00 50
    20. 20. Unit Tests Coverage in a project time frame
    21. 21. <ul><li>The main contributions of this research was obtained from the planning of a software product metrics program in a Industrial Software Factory and its implementation using a Continuous Integration environment for automatic metrics extraction from source codes. </li></ul><ul><li>GQ(I)M allowed to ensure during the planning that all collected metrics have matched measurement objectives. </li></ul><ul><li>The practices of Measurement and Analysis (MA) of CMMi level 2, was used to successfully plan the metrics program by considering which kind of analysis could be done after its implementation. </li></ul>
    22. 22. <ul><li>In a case study, software metrics were systematically collected from the source code and consolidated in a DW through the customization of a proposed CI environment strategy. This application has allowed software product analysis by using high-performance architecture, DW, and user-friendly interface with OLAP. </li></ul><ul><li>There were some created Indicators like KPI and statistical graphics that have also allowed a higher level of understanding of the project source code status and its evolution. At last, it was found that some KPI could also be used to compare Object-Oriented projects developed in the same platform. </li></ul>
    23. 23. <ul><li>[1] R. S. Pressman, “Software Engineering”, 6ª Edition, McGraw Hill, NY, 2006. </li></ul><ul><li>[2] CMMI, Version 1.2 - CMMI-DEV, V1.2, CMU/SEI-2006-TR-008 - Improving processes for better products , SEI - Carnegie Mellon University, Pittsburgh, 2006. </li></ul><ul><li>[3] R. E. Park; W. B. Goethert; W. A. Florac, CMU/SEI-96-HB-002 - Goal-Driven Software Measurement - A Guidebook. August 1996. </li></ul><ul><li>[4] Fowler M. Continuous Integration. Available in Accessed at October 29, 2009. </li></ul><ul><li>[5] Victor R. Basili, Gianluigi Caldeira, H.Dieter Rombach. The Goal Question Metric Approach. 5th ACM-IEEE International Symposium on Empirical Software Engineering (ISESE ’06), Rio de Janeiro, 2006. </li></ul><ul><li>[6] NDepend Metrics Definition. Available in Accessed at October 23, 2009. </li></ul><ul><li>[7] ISO/IEC 2 5000, Software Engineering – Software Product Quality Requirements and Evaluation (SQuaRE) - Guide to SQuaRE, 2005. </li></ul><ul><li>[8] ISO/IEC 25020, Software and System Engineering - Software Product Quality Requirements and Evaluation (SQuaRE) – Measurement Ref. Model and Guide, 2007. </li></ul><ul><li>[9] ISO/IEC 9126-3, Software Engineering – Product Quality - Part 3: Internal Metrics, Geneva, 2003. </li></ul><ul><li>[10] Suryn, W.; Abran A.; and April A. ISO/IEC SQuaRE: The Second Generation of Standards for Software Product Quality,&quot; 7th IASTED International Conference on Software Engineering and Applications, California, USA, 2003. ] </li></ul>