Presentation about Software Quality Assurance and the use of the U-QASAR methodology to create a Quality Model that reflects the company priorities and needs

1. FP7-ICT-2011-8. Project #: (318082)
How to define Quality Models for
Measuring Software Quality

2. FP7-ICT2011-8. Project #: (318082)
• A few words about speakers’ biographies and companies.
• The Project
• The Methodology
• Support to the SW development life cycle
• How does it work?
• Applying the Method in Agile
• Other uses of the Methodology

3. FP7-ICT2011-8. Project #: (318082)
The Project (1)
• An EU funded research project under FP7.
• 8 organizations from Finland, Norway,
Germany, Luxembourg and Spain: Experts in
SW Quality, Ontology modeling and
implementation, SW Engineering methods,
and SW Measurement.

4. FP7-ICT2011-8. Project #: (318082)
The Project (2)
• Objective: Create a solution to provide project
managers, software engineers, testers and
other actors with an objective measurement
of quality for:
– Software Development processes
– Software Products

5. FP7-ICT2011-8. Project #: (318082)
The Methodology (1)
• As part of the Project a Methodology has been
created:
– For gathering and exploiting data about the progress and
quality of software development projects and products.
– Covering the whole software development life cycle.
– Easy to apply in various types of IT Companies, SMEs or
large enterprises.
– May by applied as stand-alone approach for SW QA as well
as supported by quality management tools.
– Public: 1st and 2nd version are already shared online (visit
link).

6. FP7-ICT2011-8. Project #: (318082)
The Methodology (2)
• An overview of the Methodology areas:

7. FP7-ICT2011-8. Project #: (318082)
The Methodology (3)
• A Model-based Methodology:
– How to identify Quality Objectives.
– How to plan a Measurement Program: definition
of Quality Indicators and Measurement.
– How to select the metrics (by means of GQM).
– How to collect data (auto/manual).
– How to assess the validity of the measurement.
– How to provide valuable feedback out of the
measurements.

8. FP7-ICT2011-8. Project #: (318082)
• (Acronym) solution is:
– Paradigm agnostic!
– Agile or traditional LCs (V, waterfall, RUP…)
– Adaptable to a volatile environment.
Support to SW development LC (1)

9. FP7-ICT2011-8. Project #: (318082)
Support to SW development LC (2)
Requirements
Implementation Unit Testing
Validation
Design
Integration
Testing
# Redundant reqs
# Incomplete reqs
# Format errors
% Blocking cr viol.
% Critical cr viol
Complexity…
% Successful Uts
% Coverage
# Redundant specs
# Incomplete specs
# Format errors
% Successful TCs
% Coverage
# Misconceptions
% Successful TCs
% Coverage
# Misconceptions
• What to measure?

10. FP7-ICT2011-8. Project #: (318082)
Support to SW development LC (3)
• What to measure?
• % Incomplete stories, i.e. Stories lacking story points, BV estimations, categorization or
classification.
• # Scheduled vs. Solved issues: Ratio of solved/closed issues per sprint.
• Technical debt: Indicator aggregating data about pending bugs, known issues, etc after
each sprint.
• % of stories accepted per sprint.
• Perception of the customer & Team enthusiasm by meand of surveys.
• # Issues identified in retrospective meetings.

11. FP7-ICT2011-8. Project #: (318082)
Support to SW development LC (4)
Requirements, Stories, Issues
Implementation & Unit Testing
Validation, Acceptance or
Integration Tests
Design
• Where to look:
Wikis, Confluence, JIRA, Greenhopper,
Trac, Doors …
SpiraTest, TestRobot, HP QA Center,
Selenium, JIRA …
Confluence, JIRA
SVN, SONARQUBE
Deployment Bamboo, Jenkins, Continuum…

12. FP7-ICT2011-8. Project #: (318082)
How does it work? (1)
• Create a Quality Model
– It can be based on existing standards or models.
– It can be an organization’s self-made model.
– It can deal with process, products, or both.
Creation of a model consists of identifying expected
values, weights and thresholds for:
• Quality Objectives, or aggregation of indicators to evaluate the
achievement of an objective.
• Quality Indicators, or aggregation of metrics used to evaluate
• Quality Metrics, or raw measurements obtained automatically or
manually

13. FP7-ICT2011-8. Project #: (318082)
How does it work? (2)
• Create Quality Objectives (QOs) or Select existing
ones.
• Select or create Quality Indicators (QIs) by
aggregating metrics.
CONTEXT AWARENESS & NEGOTIATION
• QOs’ and QIs’ weights are adapted depending on the LC stage (e.g.
In the requirements stage a bigger relevance is given to
completeness of specifications).
• Weights, Thresholds and Values are negotiated among the
participants of the QA process

14. FP7-ICT2011-8. Project #: (318082)
The Five Methods (1)
• Method 1: Quality Objectives Setting Method
Level Example of level Example of quality
objectives
Product Mobile phone
High battery life, Good
sound quality, Low
failure rate
Process
Waterfall software development process
On time, On Budget
Sub-process Testing process
High percentage of
automation, High code
coverage
Features:
Find feature
Fast search results, low
memory usage

15. FP7-ICT2011-8. Project #: (318082)
The Five Methods (2)
• Method 1: Quality Objectives Setting Method

16. FP7-ICT2011-8. Project #: (318082)
The Five Methods (3)
• Method 1: Quality Objectives Setting Method

17. FP7-ICT2011-8. Project #: (318082)
The Five Methods (4)
• Method 1: Quality Objectives Setting Method
 Step 1: Preparation and pre-
assignment
Individual assignment
 3-8 participants from different
roles
 ½ hours / participant
Individual/pair assignment
 Few hours
 Step 5: Post-workshop
finalizing activities
 Step 2: Brainstorming
 Step 3: Prioritization by voting
 Step 4: Objective elaboration
Workshop
 3-8 participants from different
roles
 4 hours / participant

18. FP7-ICT2011-8. Project #: (318082)
The Five Methods (5)
• Method 1: Quality Objectives Setting Method
Name: Easy updateability
Description: Updating the software should be quick and easy, ideally possible without
deep technical or product knowledge
Rationale: Direct cost savings related to updates. Reduced risk of errors during
updates.
Related factors: Robustness of the software, configurability, quality of installer software
Votes 6
Quality indicator 1 Updating effort
Amount of average working hours consumed by making an update for a
single customer installation.
Current level: 3 h
Target level: 15 min
Deadline: May 9, 2013
Easiness of collecting: We have it

19. FP7-ICT2011-8. Project #: (318082)
The Five Methods (6)
• Method 2: Planning a measurement program
– Identifying quality indicator types:
• Purpose of the Measurements
• Measurability (numbers, human assessment, not measurable)
• Automation of data collection
• Target of the measurement
• Frequency of the Feedback
• Organizational Level of Information Use
– Planning the use of data

20. FP7-ICT2011-8. Project #: (318082)
The Five Methods (7)
• Method 3: Defining the Metrics:
Goal/question/metric
GQM Model

21. FP7-ICT2011-8. Project #: (318082)
The Five Methods (8)
• Method 3: Defining the Metrics:
Goal/question/metric

22. FP7-ICT2011-8. Project #: (318082)
The Five Methods (9)
• Method 3: Defining the Metrics:
Goal/question/metric

23. FP7-ICT2011-8. Project #: (318082)
The Five Methods (10)
• Method 4: Data collection
– Successful measurement activities
– Manual or automatic data collection
• Method 5: Reliability and validity of the
measures
– Assessing measurement validity
– Evaluating proposed metrics

24. FP7-ICT2011-8. Project #: (318082)
Applying the Method in Agile
• The 5 Methods can be applied:
– At the beginning of the project to define the
Quality Objectives, Indicators and Metrics, Validity
and Data collection.
– At the end of Iterations to review them.
– At the Iteration planning to re-define as necessary
to adapt to changes.
– During retrospectives sessions the need of
adapting to the proposed Q Model can be
identified.
– Involving all the stakeholders.

25. FP7-ICT2011-8. Project #: (318082)
Other uses of the Methodology
• Support in the implementation and follow-up
of certifications or standards (e.g. CMMi,
TMMi, SPICE, ISO…)  How?
– Requirements of the certification model 
Quality Objectives of the model
– QIs and metrics of the QOs will indicate the
degree of fulfillment of the standard/certification
model.

26. FP7-ICT2011-8. Project #: (318082)
Many thanks !!!
facebook.com/UQASAR
twitter.com/UQASAR
linkedin.com/groups/
UQASAR-Project-4682307
www.uqasar.eu