The presentation by Claudio A. Ardagna, Ernesto Damiani, Fulvio Frati, Sergio Oltolina, Mauro Regoli, Gabriele Ruffatti, was shown during the international conference on open source software OSS 2010, which took place at Notre Dame University, Indiana, USA, from 30th May to 2nd June 2010.

1. Spago4Q and the QEST nD Model:
An Open Source Solution
for Software Performance Measurement
Claudio A. Ardagna, Ernesto Damiani, Fulvio Frati, Mauro Regoli
Dipartimento di Tecnologie dell’Informazione
Università degli Studi di Milano – Italy
Sergio Oltolina, Gabriele Ruffatti
Engineering Ingegneria Informatica - Italy

2. Outline
• QEST nD Model
• Spago4Q Open Source Platform
• Roadmap to build the Integrated Framework
• Case Study
OSS 2010 2

3. Introduction
• The availability of detailed and updated information on
development process is paramount
– To maintain the competitiveness level
– To operate in new and more challenging markets
• The scenario of integrated information is known as Business
Intelligence, and enclose all the processes and tools for data
acquisition and analysis
• A number of structured process models have been adopted
depending on the domain and size
– To collect specific knowledge about their development processes
– To strengthen the know-how in terms of more efficiency and quality
OSS 2010 3

4. Goals
• Deploy an environment that integrates models for
software performance evaluation in a more complete
and coherent framework
– Non-invasive
– Dynamic
– Configurable
– Scalable
– Based on a formal mathematical model
• Provide a solution integrating
– The QEST nD model for process performance evaluation
– The Open Source Business Intelligence platform Spago4Q
OSS 2010 4

5. The QEST nD Model - 1
• Several mono-dimensional performance models are available
in SE context that integrates individual measurements into a
single performance index
– Models are oversimplified and unable to reflect different performance
aspects
• Traditional domains (e.g., Business Modeling) exploit multi-
dimensional models
– Data derived directly from accounting systems, implying multiple
analysis viewpoints
– Complex viewpoints analyze distinct aspects of the overall process
performance
OSS 2010 5

6. The QEST nD Model - 2
• The QEST nD model is aimed at measuring software project performances
addressing multi-dimensionality and qualitative-quantitative assessment
• In the QEST model the quality can be viewed as the integration of at least
three different viewpoints
– Economical
– Social
– Technical
• User-specific dimensions could be defined to evaluate ad-hoc analysis
dimensions
L. Buglione, “Misurare il software. Quantità, Qualità, Standards e
Miglioramento di processo nell'Information & Communication Technology,” 3rd Edition, Franco Angeli, 2008
OSS 2010 6

7. Mathematical Formalization
• QEST Performance value (P) given by the
combination
– Quantitative measurements (RP-Rough
Productivity)
– Qualitative measurements, calculated as a
perception-based measurement of the overall
product quality (QF-Quality Factor)
OSS 2010 7

8. Mathematical Formalization - 2
• The basic 3-D model QEST is
represented as a geometrical regular
tetrahedron
– Corners E, S, T and the convergence
of the edges to the P vertex describe
the performance level
– Points Qe, Qs, Qt represent the
respective dimension values (Rough Productivity)
• Values calculated as the weighted sum of a list of
n distinct measures
• Represented as the distance between points and corners
• If Quality factor is applied, points Q’e, Q’s, Q’t represent an upward or
downward translation of Qe, Qs, Qt
OSS 2010 8

9. Mathematical Formalization - 3
• Performance P calculated as the distance
between the center of gravity of
the original tetrahedron and the center
of the plane (Q’e, Q’s, Q’t) section along
the tetrahedron height
• Through computational geometry, the
multi-dimensional case is described with a simple formula for global
performance value computation
OSS 2010 9

10. Spago4Q
• Spago4Q (SpagoBI for Quality) is an open source platform for the
continuous monitoring of software quality
– Multi-process multi-project monitoring
– Collection of data executed in a fully-transparent way
– Equipped with extractors specific of most-common software process
environments (IDE, workflow management, text editing, …)
• Spago4Q (www.spago4q.org) is a vertical adaptation of SpagoBI, the Open
Source Business Intelligence suite (www.spagobi.org)
• Recently adopted by European Commission –
Directorate General for Regional Policy
OSS 2010 10

11. Spago4Q Structure
OSS 2010 11

12. Spago4Q Metamodel
OSS 2010 12

13. An Integrated Environment
• The definition of a QEST nD model is a five-step procedure
fully supported by Spago4Q
• The procedure is coherent with the PMAI (Plan-Measure-
Assess-Improve) cycle:
– PLAN, defining a set of KPIs, basing on the GQM approach, and
dimensions that characterize the analysis
– MEASURE, which includes the collection of data, and the
computation of KPI values and global performance value
– ASSESS, where results are presented in dashboards and
reports
– IMPROVE, where every negative or low value is deeply
analyzed to find problems in the process
OSS 2010 13

14. 1st Step: KPIs and Model Definition
• Declaration of a complete GQM, with the definition of
– the analysis dimensions
– the concepts to measure
– the KPIs to apply to project work-products
OSS 2010 14

15. 2nd Step: Weights and Thresholds Definition
• Couple each KPI with the
respective weight
– Indicates the importance that
such a concept plays in the
dimension it belongs to
• Define the specific thresholds
– Evaluates the value with
respect to organization
policies
• Assign (if provided) the QF to
each specific dimension
OSS 2010 15

16. 3rd Step: Value Collection
• Measures are taken directly from Spago4Q data
warehouse
– The DB is filled by data automatically collected by
extractors accessing process work-products (code
package, text documents, project information, …)
• KPIs are described in terms of
– Default value
– Minimum and maximum values (for normalization)
– KPI computation algorithm
OSS 2010 16

17. 4th Step: Global Performance Computation
• Global and dimension-wise performance
indexes are computed as KPIs that take in
input configuration data and results of the
KPIs
• Dimension performance is calculated as the
weighted sums of each specific KPI
OSS 2010 17

18. 5th Step: Reports
• Sets of reports and dashboards could be
defined and configured to satisfy reporting
and managerial needs
• Spago4Q provides methods and interfaces to
directly configure and create new reports
using all the facilities provided by the open
source reporting tool Eclipse BIRT
OSS 2010 18

19. Case Studies
• Case study based on the monitoring of the complex
performance of three real projects with several
measurements in a three-month time slot
• We use a QEST nD model called Business-Service Model
and taking into consideration four specific analysis
dimensions:
1. QEST-EC: Economic performance indicator
2. QEST-RS: Resource performance indicator
3. QEST-TE: Technical performance indicator
4. QEST-CS: Customer Satisfaction performance indicator
OSS 2010 19

20. Case Studies: GQM
OSS 2010 20

21. Case Studies: KPIs
OSS 2010 21

22. Case Studies: Dashboards
• Spago4Q dashboard gives an immediate snapshot of
the situation, highlighting problems and suggesting
areas that need a process quality improvement
– Red squares indicate under-threshold performance values
OSS 2010 22

23. Spago4Q: Reports
• Reports indicate that all projects were concluded with an
excellent global performance (close to one), showing some
issues in the process that is worth analyzing
– Project3 experienced bad performances in Technological and
Economic dimensions
OSS 2010 23

24. Conclusions
• We provide a complete multi-project multi-process
performance evaluation environment combining
– The mathematical formalization of the QEST nD model
– The monitoring and reporting facilities offered by Spago4Q
• It gives the possibility to analyze the performance of
the development process from different points of view
integrating semantically different KPIs in a single
indicator
• The described QEST nD model will be made available in
the future version of Spago4Q (www.spago4q.org)
OSS 2010 24

25. Questions?
Thank you for the attention
OSS 2010 25