Improving Quality and Cost-effectiveness in Enterprise Software Application Development: an Open, Holistic Approach for Project Monitoring & Control

ICSOB 2011
2° Int. Conference on Software Business
Brussels (Belgium), June 8-10 2011

Improving Quality and Cost-effectiveness in
Enterprise Software Application Development:
an Open, Holistic Approach for Project Monitoring & Control

Ernesto Damiani, Fulvio Frati
Università degli Studi di Milano, Italy

Luigi Buglione, Sergio Oltolina, Gabriele Ruffatti
Engineering Group, Italy
2nd International Conference on Software Business
Brussels, 8-10 June 2011

Outline

• Introduction

• The Puzzle of Project Monitoring & Control

• Case study

• Conclusions & Future Works

Brussels, 8-10 June 2011 2

Need for OS SPM Tools
• Open Source Software (OSS) relevance rapidly
increased mainly for the OSS capability
– Fostering open knowledge sharing across organizations
– Reducing Total Cost of Ownership (TCO)
– Increasing Return on Investment (ROI)

• A large amount of OSS is freely available covering
plenty of informative and business goals
– However, very few OSS projects deal with goal-oriented
measurement gathering data directly from the
organization’s information systems


Our Objective

• Describe a new, more comprehensive approach to
software project management

• Propose a roadmap to set up and manage a reliable
and efficient measurement framework

• Present Spago4Q, a complete open source platform
for process monitoring and automatic data gathering


The Puzzle of Project Monitoring & Control

• Measurement plan set-up often influenced
by
– Adoption of “standard measures” following an
adoption-by-analogy approach
– Reduction of budget devoted to measurement
& monitoring, leading to a lower level of
control on the project
– Lack or scarcity of reliable data, making it hard to
bind the mean relative error to the phenomena to be predicted

• A sound analysis of informative needs, measurements,
and metrics represents the main pieces of our Project
Monitoring & Control framework

GQM Approach - 1
• Propose a three-tier decomposition, deriving
measures (M) from the related questions (Q) to
be posed for answering the information goals (G)
of the interested stakeholders
– www.gqm.nl

• GQM-based measurement frameworks are multi-faceted
supporting multiple perspectives and analysis dimensions

• the Measurement Information Model (MIM) links the
information needs to its measurable entities and related
attributes
– Refines and reinforces the basic GQM idea, stressing the central role of
information needs and the instrumental role of measures
– Ref: ISO/IEC 15939:2007, Appendix A

Measurement Information Model (MIM)


BMP Technique
• BMP (Balancing Multiple Perspective) is a technique
for determining the right number of measures to be
applied within a measurement plan
• www.semq.eu/leng/modtechbmp.htm

• Characterized by four steps:
1. Determine the dimensions of interest in the project
2. Determine the list of the most representative measures associated with each
dimension
3. Identify which other control variables might be impacted negatively (e.g.
higher quality often means greater initial costs or longer project duration);
4. Figure out the best combination of indicators to build a measurement plan
for the project.


The QEST-LIME Family
• Performance evaluation model designed to
tackle decision-making process from a
multi-perspective viewpoint
– Analyses process Quality under, at least, the Economic,
Social, and Technological dimensions
– Provides an open, multidimensional shell according to the
management objectives of each specific development project
– Expresses performance into a unique, single value, as the combination
of the specific measures

• The QEST is extended by the LIME model to manage dynamic
contexts
– Collection, monitoring, and control of multidimensional measures are
applied at each software life cycle phase
– http://www.semq.eu/leng/modtechqlm.htm


Spago4Q
• Spago4Q (SpagoBI for Quality) is
– www.spago4q.org
– An OS platform for the continuous monitoring of software
quality
– A vertical adaptation of OS Business Intelligence
suite SpagoBI,
– Recently adopted by the European Commission –
Directorate General for Regional Policy (DG-REGIO)

• Spago4Q provides
– Multi-process multi-project monitoring
– Automatic data collection executed in a fully transparent way
– Equipped with extractors specific of most-common software process
environments (IDE, workflow management, text editing, etc.)

• Based on a modular metamodel that allows to apply, at
the same time, to running projects adopting different
development process, the same measurement
framework

Spago4Q Metamodel


Spago4Q Architecture


Putting all Pieces Together

• GQM, BMP, QEST-LIME, and tools like Spago4Q provide the pieces needed
to solve the process monitoring puzzle
• Our approach proposes to join them, keeping the best from each
– Define what we need to measure (using a GQM-based technique)
– Filter and prioritize what we need to measure (using the BMP technique).
– Determine the project/activity performance values with a holistic view (using QEST-
LIME), highlighting the improvement goals to manage.
– Automate the last step with an OSS tool (e.g. Spago4Q), providing faster and transparent
data collecting

Case Study – 1
• The framework was applied to two large proprietary products
of Engineering Health Software Factory (HSF)
– projects data were analyzed and collected in the period from January
2009 to October 2010


Case Study – 2
• To monitor each phase of the HSF software life cycle, we exploit
BMP and LIME

• The main business goal for our goal-oriented analysis was to reduce
the overall production cost

• Such cost is mainly driven by three factors:
– Lack of requirement stability, as a source of overhead in design and
implementation activities
– Working group management overhead, as a source of delays and variance
in milestones
– Corrective and adaptive maintenance activities

• Cost factors allowed to define the measurement goals and the
complete list of measures, associated with the respective goal


Cost Factors

Requirements
(analysis)

Measurements w.r.t. Cost Factors
CMMI-DEV
Goal Measure Measure Id.
v1.3 PAs
REQ-E- 1 Incidence of delays on delivery milestones (deliverables) w.r.t. total # deliverables PMC – MA PR-REQ-E-M1.1
RD – REQM -
REQ-E-2 Requirements Variability PR-REQ-E-M1.2
PMC
REQ-S-1 #. detected criticalities during the human resources management w.r.t. group size in the phase PPQA – PMC PR-REQ-S-M1.1
REQ-S- 2 User satisfaction or % users involvement in the phase all PA (GP2.7 PR-REQ-S-M1.2
REQ-T- 1 Document quality: respect of quality standard PPQA PR-REQ-T-M1.1
DEV-E- 1 Incidence of delays on delivery milestones (deliverables) w.r.t. total # deliverables PMC – MA PR-DEV-E-M2.1
DEV-S- 1 # detected criticalities during the human resources management w.r.t. group size in the phase PPQA – PMC PR-DEV-S-M2.1
DEV-T- 1 Document quality: respect of quality standard PPQA PR-DEV-T-M2.1
DEV-T- 2 Software quality: complexity, compliance, maintainability PPQA – VAL PR-DEV-T-M2.2
DEV-T- 3 Compliance with end-phase checklist PPQA PR-DEV-T-M2.3
TES-E- 1 # Defects detected before System Test by PR or verifications w.r.t. code size VER – VA – PMC PR-TES-E-M3.1
TES-E- 2 Incidence of delays on delivery milestones (deliverables) w.r.t. total # deliverables PMC – MA PR-TES-E-M3.2
TES-S-1 N.o. detected criticalities during the human resources management w.r.t. group size in the phase PPQA – PMC PR-TES-S-M3.1
Incidence of the n.o. reviews (peer reviews or inspection reviews) w.r.t. total # deliverables PPQA – VER PR-TES-T-M3.1
TES-T- 1
Percentage defects distribution on phases that produced the defects (consider only analysis and design phase) PP – PMC – MA PR-TES-T-M3.2

TES-T- 2 Compliance with end-phase checklist PPQA – PMC PR-TES-T-M3.3
Incidence of defects tested in running and testing phase w.r.t. maintained code size (Lines of Code or Function
PMC – MA PR-MAN-E-M4.1
MAN-E- 1 Points)
Mean defect resolution time w.r.t. severity during running phase IRP PR-MAN-E-M4.2
MAN-S- 1 N.o. detected criticalities during the human resources management w.r.t. group size in the phase PPQA – PMC PR-MAN-S-M4.1
MAN-S- 2 User satisfaction all PA (GP2.7) PR-MAN-S-M4.2
MAN-T-1 Percentage defects distribution on phases that produced the defects (consider only analysis and design phase) PP – PMC - MA PR-MAN-T-M4.1

Results analysis - 1
• The analysis of measures results suggests different
improvement actions to monitor each phase, to
improve the overall economic results
– Increase document quality produced in the requirements
phase (PR-REQ-T-M1.1)
– Increase software quality to facilitate its maintainability in
the development phase (PR-DEV-T-M2.1 and PR-DEV-T-
M2.2)
– Increase the number of reviewed deliverables (PR-TES-T-
M3.1)
• The reviews performed in the analysis and design phases had the
goal to discover bugs early in the development cycle (PT-MAN-T-
M4.1, PR-TES-T-M3.2)


Results analysis - 2
Product 1 Product 2
Goals Measure Year 2009 2010 2009 2010
Economic Dimension
Reduce Defects resolution costs 0.7000 0.7550 0.7870 0.8530
PR-MAN-E
PR-MAN-E-M4.1 0.5244 0.5099 0.5793 0.7319
PR-MAN-E-M4.2 0.8173 0.9244 0.9262 0.9333
Reduce criticality in working group
Social Dimension
management 0.6810 0.7000 0.7100 0.6850
PR-MAN-S
PR-MAN-S-M4.1 0.6944 0.5833 0.5000 0.3000
PR-MAN-S-M4.1 0.6750 0.7500 0.8000 0.8500
User satisfaction Technical Dimension
0.6304 0.6552 0.7005 0.6842
PR-MAN-T
Increased #. of checks/reviews PR-MAN-T-M4.1 0.6304 0.6552 0.7005 0.6842

• Analysis of the goal Reduce Defects Resolution Cost in the
Running phase
– Improvement of the 7.2% for Product 1 and 7.6% for Product 2
– Improvement of Mean Defect Resolution Time measure (PR-MAN-E-
M4.2)
– Consequent improvement of Social (S) dimension, and in particular the
User Satisfaction metric (PR-MAN-S-M4.2)
• Received benefit from the improvements of document quality and the reduction
of Mean Defect Resolution Time


Results Analysis – 3
Product 1 Product 2
Goals Measure Year 2009 2010 2009 2010
Requirement phase
Compliance with pre-defined documents q.l. 0.8333 0.8667 0.8667 0.9000
PR-REQ-T-M1.1
Development phase
Compliance with pre-defined documents q.l. 0.8333 0.8667 0.8667 0.9000
PR-DEV-T-M2.1
Development phase
Compliance with pre-defined software q.l. 0.5000 0.6000 0.7500 0.9000
PR-DEV-T-M2.2
Test phase
Increase n.o. checks/reviews 0.7500 0.8750 0.5250 0.6250
PR-TES-T-M3.1

• Analysis of measures correlated to the Reduce Defects
Resolution Cost goal
– The increasing of the number of reviews (PR-TES-T-M3.1) had a
positive effect on the improvement of the overall quality of
documents (PR-REQ-T-M1.1 and PR-DEV-T-M2.1)
– The improvement of software quality (PR-DEV-T-M2.2 w.r.t. PR-
MAN-T-M4.1) was mainly due to the decreasing the percentage of
defects in initial development phases (analysis and design)


Results Analysis – 4


Conclusions
• We present our experience in developing an integrated
framework for process monitoring integrating
– GQM approach
– BPM technique
– QEST-LIME framework
– Spago4Q platform

• The effectiveness of the framework has been tested on
two on-going projects, and an analysis of the achieved
results has been resented

• Our approach represents a good starting point for a full
implementation of a Balanced Scorecard (BSC) technique

Future Works

• Future works will focus on
– the improvement of Spago4Q reporting features
– implementation of a GQM(R) matrix for choosing new
possible measures in order to cover a larger plateau of
information needs at the same cost

• The framework is available as a demo on the
Spago4Q web site (www.spago4q.org)


Questions?

THANKS FOR YOUR ATTENTION!

Contacts:

http://sesar.dti.unimi.it
http://www.spagoworld.org

fulvio.frati@unimi.it

Improving Quality and Cost-effectiveness in Enterprise Software Application Development: an Open, Holistic Approach for Project Monitoring &amp; Control

More Related Content

Similar to Improving Quality and Cost-effectiveness in Enterprise Software Application Development: an Open, Holistic Approach for Project Monitoring &amp; Control

More from Luigi Buglione

Recently uploaded

Improving Quality and Cost-effectiveness in Enterprise Software Application Development: an Open, Holistic Approach for Project Monitoring &amp; Control

Improving Quality and Cost-effectiveness in Enterprise Software Application Development: an Open, Holistic Approach for Project Monitoring & Control

Similar to Improving Quality and Cost-effectiveness in Enterprise Software Application Development: an Open, Holistic Approach for Project Monitoring & Control

Improving Quality and Cost-effectiveness in Enterprise Software Application Development: an Open, Holistic Approach for Project Monitoring & Control