The presentation supported the speech by Gabriele Ruffatti (founder of the SpagoWorld initiative) at SIMPDA 2014 (Milan, Italy - November 19-21, 2015). The presentation focuses on the innovative approach named Productivity Intelligence supported by Spago4Q - the open source analytic of SpagoBI suite for Quality and Performance Improvement- that allows companies and organizations to effectively monitor performances, improve quality practices and achieve higher capability levels. www.spagoworld.org

1. Measuring quality of developments
in a large industrial software factory
www.eng.it
A living story with Open
Source Software
Gabriele Ruffatti, Director
Architectures & Consulting Services
Big Data & Open Source Competency Centers
Methodologies, Processes & Services for
Engineering's Software Labs
Engineering Group

2. ( Wilmington
) São Paulo / Rio de Janeiro / Recife Belo
Business integration
Consulting
Outsourcing
Products and solutions
( ) ( Buenos Aires ) ( Brussels )
Horizonte / Curitiba
USA BRAZIL ARGENTINA BELGIUM
TECHNOLOGICAL SOLUTIONS
25
mn€/year
2 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it
( Belgrade )
REPUBLIC OF
SERBIA
A global player
31 branches in ITALY
7.2%
1,000
about 7,300 Large accounts
Professionals
822.8 mn€
Italian market
PRODUCTION
& INNOVATIVE
APPLICATIONS
RESEARCH
IDEAS FOR
RESEARCH PROJECTS
INNOVATION
EXPERIMENTAL
CHECKS
RESEARCH PROJECTS
RESULTS
I N V E S T M E N T S
in I N N O V A T I O N
Participation in European
research programs and creation
of a network of collaborations
+
ENGINEERING GROUP

3. Economic efficiency
OPEN SOURCE
Technical efficiency
Strategic efficiency
Social efficiency
3 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

4. OPEN SOURCE
Source: OW2, Cédric Thomas, 2014
4 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

5. INTEGRATOR
OPEN SOURCE @Engineering
knowledge sharing
INNOVATOR
collaborative projects
Competitive lever
PURE PLAYER
global communities
DIGITAL AGENDA
FOR EUROPE
5 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

6. OPEN SOURCE @Engineering INTEGRATOR
System integrators DO NOT sell “licenses”
but skills and know-how
Cost reduction
Flexibility
Innovation
Knowledge as a Common
6 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

7. OPEN SOURCE @Engineering PURE PLAYER
www.spagoworld.org
GLOBAL COMMUNITIES
7 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

8. OPEN SOURCE @Engineering SPAGOBI
A comprehensive business
intelligence suite
Innovative themes and
solutions
100% open source software forever
user-oriented, flexible and scalable
8 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

9. OPEN SOURCE @Engineering SPAGOBI
9 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

10. Cloud Computing
Big Data
Future Internet
Privacy/Security
OPEN SOURCE @Engineering INNOVATOR
10 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

11. ENGINEERING GROUP ORGANIZATION
ENGINEERING
TECHNICAL
UNIT
SOFTWARE LABORATORIES
COMPETENCY CENTERS
• Automation Control
• BI DataWarehouse
• ECM
• ERP
• GIS
• Managed Operations
• Mobile
• Big Data
• Open Source SpagoBI
R D
BUSINESS
UNITS
PA HEALTHCARE
TELCO UTILITIES
INDUSTRY SERVICES
FINANCE
MARKETS
11 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it 11

12. I want to know the productivity
of my software factory.
Which is corporate audit results ?
Is there REALLY
a way to measure performance ?
Which are
How can I improve
the development process ?
Is my project on track?
12 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it
Which is the
quality level of my product ?
users' and customers'
level of satisfaction ?
How productive
is my organization ?
How can I improve
performance?
How can I compare
different labs?
Top
Manager
Quality
Manager
Project
Manager
REQUIREMENTS MANAGERS’ NEEDS

13. REQUIREMENTS COMPLIANCE TO QUALITY STANDARDS
● Continuous Quality Improvement in Engineering Group's
projects
● Unified Infrastructure supporting quality processes granting
flexibility and adaptability to Engineering's Software Labs
● CMMI-DEV and ISO certifications, as independent method to
validate the compliance of processes and infrastructures
with quality standards
13 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

14. BACKGROUND MODEL TOOL
QEST nD model, a conceptual framework for measuring process
performance based on multiple analysis dimensions
Spago4Q, the open source SpagoBI analytic for Quality and
Performance Improvement
14 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

15. BACKGROUND QEST MODEL
Method: Performance is expressed as the combination of the specific ratios
selected for each of the 3 dimensions of the quantitative assessment
(Productivity - PR) and the perceived product quality level of the qualitative
assessment (Quality - Q)
Performance = PR + Q
Model: QEST (Quality factor + Economic,
Social Technical dimensions) is a
“structured shell” to be filled according
to management objectives in relation to
a specific project
Such a model has the ability to handle
independent sets of dimensions without
predefined ratios and weights - referred
to as an open model
Source: Buglione L. Abran A., QEST nD: n-dimensional extension and generalisation of a Software Performance Measurement Model, International Journal of
Advances in Engineering Software, Elsevier Science Publisher, Vol. 33, No. 1, January 2002, pp.1-7
15 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

16. BACKGROUND QEST MODEL
Target: measuring project performance (p) using 3 distinct viewpoints
Input Data: list of weighted ratios for each dimension and quality
questionnaires
Output Data: an integrated normalized value of performance
It is possible to measure performance considering at least 3 distinct geometrical
concepts:
• Distance between the tetrahedron base
center of gravity and the center of the
plane section along the tetrahedron
height – the greater the distance from 0,
the higher the performance level;
• Area of the sloped plane section – the
smaller the area, the higher the
performance level;
• Volume of the lowest part of the
truncated tetrahedron – the greater the
volume, the higher the performance level.
16 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

17. BACKGROUND SPAGO4Q
17 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

18. THE PROCESS PMAI APPROACH
The procedure is coherent with the PMAI (Plan-Measure-Assess-Improve)
cycle:
PLAN, defining a set of metrics, based on the GQM approach, and
possible dimensions of analysis characterizing the analysis
MEASURE, including the collection of data, and the computation of
metric values and global performance value
ASSESS, presenting results through dashboards and reports
IMPROVE, analyzing in detail each value below expected thresholds in
order to find possible problems or bottlenecks from a process based
viewpoint
18 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

19. THE PROCESS S1. METRICS MODEL DEFINITION
Declaration of a complete GQM, with the definition of
the analysis dimensions
the concepts to measure
the metrics to apply to project’s work-products
19 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

20. THE PROCESS S2. WEIGHTS THRESHOLDS DEFINITION
Couple each metric with the
respective weight
Indicates the importance that
such a concept plays in the
dimension it belongs to
Define the specific thresholds
Evaluates the value with respect
to organization policies
Assign (if provided) QF to each
dimension
Give to each dimension a quality
evaluation
20 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

21. THE PROCESS S3. DATA GATHERING
Measures are taken directly from Spago4Q data warehouse
The DB is filled by data automatically collected by extractors
accessing process work-products (code package, text documents,
project information, …)
Metrics are described in terms of:
Name of the model to which the metric is assigned to
Default value
Minimum and maximum values (for normalization)
KPI computation algorithm
21 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

22. THE PROCESS S4. PERFORMANCE CALCULATION
Overall and dimension-wise performance indexes are computed as
KPIs that take in input configuration data and results of the
metrics
The performance value of each dimension is calculated as the
weighted sum of each selected measure by its assigned weight
for that dimension
22 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

23. THE PROCESS S5. REPORTING
Sets of reports and dashboards could be defined and configured
to satisfy reporting and managerial needs
Spago4Q provides methods and interfaces to directly configure
and create new reports using all the facilities provided by the
SpagoBI suite of analytical tools
23 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

24. CASE STUDY AM IN FINANCE
Application Management services
Software Maintenance (Corrective, Adaptive, Perfective, Preventive) for a large mission-critical
system in a Finance Institute
Services started in 2006 (analysis period : January 2008 – June 2010)
Verify QEST nD applicability and results in a context of AM Services
Define a QEST nD model aligned to the AM services goals
Monitor the effectiveness of improvement action with specific goals and metrics
Goals
EC-G3 Reduce the rework (intended as impact of defects in UAT or production
environment)
TE-G1 Improve the deploy process
TE-G5 Improve effectiveness of peer reviews
Improvement actions
Deploy process automation and automatic analysis of source code
Progressively increasing of the number of peer reviews on critical work products
Specific tasks were included in Impact analysis phase at the aim to:
Classify and identify critical Work Products to be reviewed
Assign an owner to solve complex defects impacting on different development
streams
Root-cause analysis of the recurring defects
24 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

25. CASE STUDY DIMENSIONS OF ANALISYS
Four analysis dimensions:
1. Economical (EE)
2. Technical (TT)
3. Resource Usage (RRSS)
4. Customer Satisfaction (CCSS)
Each dimension is characterized by a specific metrics set for
process evaluation
Performance values for each dimension allow to identify process
areas that need improvements
25 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

26. CASE STUDY GOALS METRICS
Four analysis dimensions and goals as follows:
1. Economical (EE)
E.G1 Reduce the effort of corrective maintenance
E.G2 Improve the number of delayed deliverables
E.G3 Reduce the rework (intended as impact of defects in UAT or production environment)
2. Technical (TT)
T.G1 Improve the deploy process
T.G2 Reduce the resolution time for defects and technical issues
T.G3 Improve quality of documents and source code
T.G4 Reduce the rework (intended as impact of defects during development phase)
T.G5 Effectiveness of peer reviews
T.G6 Improve non regression test
3. Resource Usage (RRSS)
RS.G1 Reduce impact of human resource management issues
RS.G2 Improve hardware system availability
4 Customer Satisfaction (CCSS)
CS.G1 Improve user satisfaction about training courses and application services
26 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

27. CASE STUDY GOALS METRICS
Dimension Metric Description Formula
Economical (E) Incidence of Corrective Maintenance Effort w.r.t. maintained code size Corrective Maintenance Effort/ KLOC
Ratio Corrective Maintenance Effort - Adaptive Maintemance Effort Corrective Maintenance Effort/ Adaptive
Maintenance Effort
Incidence of Delayed Deliverables w.r.t. total number of Deliverables no. Delayed Deliv. / no. Deliv.
Incidence of Defects after system test w.r.t. total number of Defects no. Defects in UAT or production / total no. of
Defects
Resource Usage
(RS)
Human Resources management issues w.r.t. total number of issues admitted for
working group size
no. HR issues / no. Issues for group size
Hardware System Availability Percentage System Availability
Technical (T) Technical management issues w.r.t. total number of issues admitted no. Technical issues / no.issues admitted
Issues Mean Resolution Time Total Res. Time / no. Issues
Document quality: respect of document quality standard Percentage of positive response to a
checklist
Software Complexity Results of automatic static code analysis
Coding rules non-conformity level Results of automatic static code analysis
Software Maintenability Results of automatic static code analysis
Incidence of Peer Reviews w.r.t. total number of Deliverables no. Peer reviews / no. Deliverables
Number of Defects discovered by peer reviews w.r.t. total number of Defects no. Peer review defects / total no. defects
no. Defects / FP
Incidence of Defects Due to Design Phase w.r.t. total number of Defects no. Defects(Design phase) / Total no.
Defects
for any phase p
Test coverage w.r.t. Requirements no. Test Cases / no. Requirements
Production Defects Mean Resolution Time Total Res. Time / no. defects
Customer
Satisfaction (CS)
Training Services Questionnaire results
User Satisfaction Questionnaire results
27 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

28. CASE STUDY RESULT – QEST DASHBOARD
28 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

29. CASE STUDY RESULT – DIMENSIONS TREND ANALYSIS
Last results for each dimension
Trend for each dimension
29 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

30. CASE STUDY RESULT – GLOBAL TECHNICAL % INCREASE
30 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

31. CASE STUDY RESULT – SAMPLE
[AM-EC-M.04] Defects reduction in UAT and production environment
[AM-TE-M.11] Defects mean resolution time reduction
31 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

32. CASE STUDY RESULT – SAMPLE
[AM-TE-M.01] Technical issues reduction: specifically related to deployment process
[AM-TE-M.02] Technical issues mean resolution time
32 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

33. CASE STUDY RESULT – SAMPLE
[AM-TE-M.07] Number of peer reviews actually executed vs. number of critical Work
Products
[AM-TE-M.08] Defects or potential defects discovered during peer reviews
[AM-TE-M.09] Incidence of defects due to design phase
33 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

34. ENGINEERING’S SOFTWARE LABS TECHNICAL INFRASTRUCTURE
34 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

35. ENGINEERING’S SOFTWARE LABS SOFTWARE INFRASTRUCTURE
Service Management Application Lifecycle Management
Program
Management Requirement
Collaboration
Forum, Blog, Wiki
Service
desk
35 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it
Document
management
Problem
Management
Change
Management
Request
Management
Risk
Management
Management
Development
Management
Software
Quality
Test
Management
Deploy
Management
Repository
Documenti e KB
Configuration
Management
Sistema di Reporting e
Cruscotti SLA
Knowledge Management
System Monitoring
AL
M
SCM
IDE
Continuous
Integration
Test
Automation
Agile Project
Management
Monitoring Control
Software Factory
CMDB
Incident
Management
Catalogo Riuso
Siti cliente
Customer
Satisfaction

36. PRODUCTIVITY INTELLIGENCE INFRASTRUCTURE
I want to know the productivity
of my software factory.
Quality emerge ad the result of three dimensions of
analysis:
Economic (EE)
Social (SS)
Technical (TT)
Performance values for each dimension
allow to identify process areas that need
improvements
36 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

37. PRODUCTIVITY INTELLIGENCE SOCIAL ANALYSIS
● Social Dimension is a First Class Citizen
● Quantitative data about how people adhere
to corporate processes
● Qualitative data from LimeSurvey about
satisfaction level of customers, integrators,
developers
● Net Promoter Score approach
37 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

38. PRODUCTIVITY INTELLIGENCE DRILL DOWN VIEWS
Top Manager (TM)
Level 1
ESL Chief Manager
Level 2
ESL Lab Manager
Level 3
Project Manager (PM)
ESL
ESL 1 ESL 2 ESL 3
PRJ 1
PRJ n
PRJ 1
PRJ n PRJ n
38 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it
PRJ 1
Engineering's Software Labs
Project Development Project Development Application Maintenance

39. PRODUCTIVITY INTELLIGENCE GENERAL DASHBOARD
39 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

40. PRODUCTIVITY INTELLIGENCE QEST ANALYSIS
40 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

41. PRODUCTIVITY INTELLIGENCE QEST DASHBOARD
41 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

42. PRODUCTIVITY INTELLIGENCE SINGLE INDICATOR
42 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

43. PRODUCTIVITY INTELLIGENCE ANSWERS TO MANAGERS’ NEEDS
Productivity Intelligence
enables performance
improvement!
Finally we can REALLY
measure performance!
Now I know how productive
my organization is!
Now I can compare
Labs performance!
Users Customers
feedbacks are now integrated
with corporate data!
Through audit dashboards,
corporate QA
is under control !
43 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it
I can monitor the
quality level
of my product !
I know if my project is
on track I can identify
issues !
The development process
is under control and
I can improve it !
Top
Manager
Quality
Manager
Project
Manager

44. PRODUCTIVITY INTELLIGENCE RESULTS NEXT STEPS
INTEGRATOR
Skill consolidation SW FACTORY EFFECTIVENESS
New market sector: ALM, PRJ AUTO
INNOVATOR
PURE PLAYER
Product
Improvement
Collaborative projects
Research developments
Projects: RISCOSS
Conferences, publications (ISSRE, IT Confidence, ICSOB)
44 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it

45. PRODUCTIVITY INTELLIGENCE CONCLUSIONS
OPEN SOURCE HAS NOT INTRINSIC VALUE PER SE
45 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it
45
Prepare the environment and build the ecosystem
Stimulate creativity
Help bring innovation into market
Deliver market-ready offerings
Measure, assess, and value the results
LET’S MAKE IT HAPPEN!

46. Thanks for your Attention !
We care of your problems and we have in mind a solution
resources: www.spago4q.org
ecology of value: www.spagoworld.org/blog/
comments: www.linkedin.com (group: SpagoWorld)
www.twitter.com (@gruffatti, @spagoworld, @engineeringspa)
contacts:
mailto: gabriele.ruffatti@eng.it
46 SIMPDA 2014 – Milan,Italy November 20th, 2014 www.eng.it