The document contains information about errata in course materials on software estimation, an upcoming lecture on software project management with opportunities for student presentations, and slides from a lecture on quality in software project management that discuss setting quality goals and mapping quality practices to goals and risks.

1. SoberIT
Software Business and Engineering Institute
Errata!
 “Estimation of Software” has a bug!
 “The multiplier for high complexity and external output type
is higher than the multiplier for high complexity and external
input type.”
 Instead, you should look up the values from the table for
external inquiry type, as with other user types
 The reason:
 Some descriptions of Albrecht function point method have
instructed to do as described in the material, but this
suggestion has been changed later.
HELSINKI UNIVERSITY OF TECHNOLOGY

2. SoberIT
Software Business and Engineering Institute
Lecture on February 16th
 Student presentations!
 Prepare a presentation of 15-20 minutes on topic of your interest
(related to software project management), and get 5 lecture
summary points
 And/or
 Attend the session, participate in discussions and share your own
experiences, and earn max 5 lecture summary points by writing
a summary of the presentations & discussion
 If you want to give a presentation, please send email to
tuomas.niinimaki@tkk.fi
HELSINKI UNIVERSITY OF TECHNOLOGY

3. SoberIT
Software Business and Engineering Institute
T-76.5612 Software Project Management
5: Quality in Software Project Management
Tuomas Niinimäki
(many slides by Juha Itkonen & Mika Mäntylä)
Helsinki University of Technology
HELSINKI UNIVERSITY OF TECHNOLOGY

4. SoberIT
Software Business and Engineering Institute
A Plan
HELSINKI UNIVERSITY OF TECHNOLOGY 4

5. SoberIT
Software Business and Engineering Institute
The Reality
HELSINKI UNIVERSITY OF TECHNOLOGY 5

6. SoberIT
Software Business and Engineering Institute
What quality practices can do for my
project?
 Provide headlights (Information on quality)
 Where are we?
 … and what lies ahead?
 Enable changing our plan
 We can get there, but later than planned
 We can get almost there and stick to our
schedule
 But if we run full steam ahead according
to our plan we will definitely crash
HELSINKI UNIVERSITY OF TECHNOLOGY 6

7. SoberIT
Software Business and Engineering Institute
Contents
 Setting quality goals
 From goals to quality practices
 Quality information in project planning and tracking
HELSINKI UNIVERSITY OF TECHNOLOGY 7

8. SoberIT
Software Business and Engineering Institute
Quality Goals
HELSINKI UNIVERSITY OF TECHNOLOGY 8

9. SoberIT
Software Business and Engineering Institute
Strive for good quality
Bad quality leads to rework,
higher costs, delays, and
problems in operation
HELSINKI UNIVERSITY OF TECHNOLOGY 9

10. SoberIT
Software Business and Engineering Institute
What is quality (in software)?
HELSINKI UNIVERSITY OF TECHNOLOGY

11. SoberIT
Software Business and Engineering Institute
ISO 9126 Quality attributes
 Functionality  Efficiency
 Suitability  Time behavior
 Accurateness  Resource behavior
 Interoperability  Maintainability
 Security  Analyzability
 Reliability  Changeability
 Maturity  Stability
 Fault tolerance  Testability
 Recoverability  Portability
 Usability  Adaptability
 Understandability  Installability
 Learnability  Conformance
 Operability  Replaceability
 Attractiveness
+ suggested metrics for each quality attribute
HELSINKI UNIVERSITY OF TECHNOLOGY 11

12. SoberIT
Software Business and Engineering Institute
QUALITY
PRODUCT PEOPLE PROCESS
HELSINKI UNIVERSITY OF TECHNOLOGY

13. SoberIT
Software Business and Engineering Institute
QUALITY
PRODUCT PEOPLE PROCESS
 Product is error-free  The expectations of  Control and
the customer or user management
 Product meets the have been met
specifications  Expectations,
 Someone gets value estimates, forecasts
 Product has certain from the result
measurable attribute  What is value?  Reproducibility
Differs depending
 e.g. durability, on who you ask:
weight, SMS
support
developer, user,
tech. support
 Improvement
 Learning from
mistakes and
 Sustainability successes
 Conformity /
commitment
HELSINKI UNIVERSITY OF TECHNOLOGY

14. SoberIT
Software Business and Engineering Institute
Quality goals
HELSINKI UNIVERSITY OF TECHNOLOGY

15. SoberIT
Software Business and Engineering Institute
Setting the target - Quality goals
 Generic quality characteristics are not enough
 Customer’s requirements and needs
 Product characteristics
 Application domain
 Development technologies and environment
 Project characteristics
 Specific quality goals are needed
 Specific goals for this project
 Unambiguous
 Connected to a concrete project, product, and problems
we need to define precisely what qualities we require of a system
HELSINKI UNIVERSITY OF TECHNOLOGY 15

16. SoberIT
Software Business and Engineering Institute
From a general to a specific quality goal: An Example
 Adaptability
 Attributes of software that bear on the opportunity for its
adaptation to different specified environments without
applying other actions or means than those provided for this
purpose for the software considered.
 Adaptability
 The MobWebClient software must work with all advertised
compatible browser and e-mail environments, with or without
java, without specific configuration.
HELSINKI UNIVERSITY OF TECHNOLOGY 16

17. SoberIT
Software Business and Engineering Institute
Quality goals
 Quality goals point out the important quality characteristics for
 the project,
 the product, and
 the whole organization
 Quality goals
 show where to focus effort and resources
 help selecting suitable practices and methods
 steer every activity of the project into the right direction
 Quality goals can reflect different qualities for different parts of
the system
HELSINKI UNIVERSITY OF TECHNOLOGY 17

18. SoberIT
Software Business and Engineering Institute
Focusing the QA efforts – Quality Risks
 Risk – a potential threat to projects objectives
 With uncertain probability
 Quality goals tell us what are the important qualities to achieve
in this project
 Quality risk analysis focuses our QA efforts to the most
important things
 E.g. if functional correctness is one of our quality goals:
What are the most important functions that we should test
thoroughly vs. all the functions we could test?
HELSINKI UNIVERSITY OF TECHNOLOGY 18

19. SoberIT
Software Business and Engineering Institute
What we should test vs. what we could test?
Reduced value due to defects
Cost of finding defects
HELSINKI UNIVERSITY OF TECHNOLOGY

20. SoberIT
Software Business and Engineering Institute
Example – Risk analysis matrix
Characteristic Impact Probability Exposure
Usability 3 4 12
Functional 2 2 4
correctness
Conformance 1 3 3
to standard X
HELSINKI UNIVERSITY OF TECHNOLOGY

21. SoberIT
Software Business and Engineering Institute
Example – Statistical Risk Analysis Matrix
Impact * Probability = Re
New Design Com- Weigh. Risk
Dev Cust. Avg Func. Qual. Size plexity Sum Exposure
Weight 5 5 1 3
Interest
Calculation 3 3 3 2 3 3 3 37 111
Close
Account 1 3 2 2 2 2 3 31 62
Create 2 1 1,5 3 3 2 3 41 61,5
Account
The numbers and calculations are not important – Idea
is to get the features into priority order Modified slide, originally from Ståle Amland
HELSINKI UNIVERSITY OF TECHNOLOGY 21

22. SoberIT
Software Business and Engineering Institute
Prioritising risks (I x P = E)
 I = Impact, P = Probability, E = Exposure
 What does the exposure number mean?
 Scale of a risk
 Used to rank risks
 high I x low P = low I x high P
 Risks with very severe impacts can have average exposure
 Usually impact is easier to estimate than probability
 Can lead to low exposures for high impact risks
HELSINKI UNIVERSITY OF TECHNOLOGY 22

23. SoberIT
Software Business and Engineering Institute
Prioritising risks
 Goal is to prioritize
 Which risks deserve most attention
 The discussion about the severities and probabilities is probably
more valuable than the exact ranking
 In QA, prioritization means distribution of efforts, not order of
actions
 Less risky application areas cannot be ignored
 Requirement priority is not test priority
HELSINKI UNIVERSITY OF TECHNOLOGY

24. SoberIT
Software Business and Engineering Institute
Quality Practices
HELSINKI UNIVERSITY OF TECHNOLOGY 24

25. SoberIT
Software Business and Engineering Institute
Quality practices
 Many practices strive for better quality
 Testing
 Reviews, inspections
 Conventions, guidelines
 Documenting
 Design and development methods & techniques
 Project management practices
HELSINKI UNIVERSITY OF TECHNOLOGY 25

26. SoberIT
Software Business and Engineering Institute
Quality practices
 Plan which practices are used to achieve and measure each of
the quality goals
 Applying existing practices already in use
 Learning best practices from other projects and organizations
 Improving and combining existing practices
 Inventing new practices
 Quality goals give guidance for performing the practices
HELSINKI UNIVERSITY OF TECHNOLOGY

27. SoberIT
Software Business and Engineering Institute
Mapping the goals and practices
Quality goals Quality threat
Practices
Functional Conformance Usability Performan DB changes One developer
correctness to standard X ce break existing has no
client sw experience in
J2EE
Functional
testing 3 1
GUI
prototyping 2
Code reviews
3
3 = good practice
2 = helps
1 = might help
HELSINKI UNIVERSITY OF TECHNOLOGY 27

28. SoberIT
Software Business and Engineering Institute
Mapping the goals and practices
Quality goals Quality threat
Practices
Functional Conformance Usability Performan DB changes One developer
correctness to standard X ce break existing has no
client sw experience in
J2EE
Functional
testing 3 1
GUI
prototyping 2
Code reviews
3 1 2
Regression
testing client 1 3
sw
 Documents how quality goals are to be achieved
 Focuses activities into right issues
 Reveals and communicates
 explicitly identified threats
 quality goals with weak practices
28
HELSINKI UNIVERSITY OF TECHNOLOGY

29. SoberIT
Software Business and Engineering Institute
How to measure the quality goals
 Some quality characteristics can be measured
quantitatively
= Hard metrics
 e.g. features passing tests, performance
 Some can be assessed subjectively
= Subjective evaluation
 e.g. usability, conformance to a standard
 Some cannot be measured at all (in a practical
situation)
= Indirect metrics
 e.g. maintainability, reusability
HELSINKI UNIVERSITY OF TECHNOLOGY 29

30. SoberIT
Software Business and Engineering Institute
Evaluating and tracking quality - Hard metrics
 Hard metrics are concrete and clear
 If the goal can be quantified, do so
 What measures are used?
 Who, how and when?
 How the measures are used?
 Numbers are easy to present and understand
 Easy to track and observe changes and trends
 Easy to misinterpret
 E.g. what can we infer based on the number of found
defects alone?
 Historical data helps in detecting trends
 Characteristics of defects (severity, technical type)
HELSINKI UNIVERSITY OF TECHNOLOGY 30

31. SoberIT
Software Business and Engineering Institute
Evaluating and tracking quality - Subjective evaluation
 Subjective evaluation as a measure
 Define how the goal is assessed
 What are the metrics
 Who, how and when
 How the results are used
 Reviews and assessments
 Subjective opinion
 Qualitative data
 E.g., subjective assessment of quality of a video stream
 Tracking not as straightforward -> must quantify
 E.g., compare to reference video streams of different
quality
HELSINKI UNIVERSITY OF TECHNOLOGY 31

32. SoberIT
Software Business and Engineering Institute
Evaluating and tracking quality - Indirect metrics
 Define how to achieve, evaluate and track these goals too!
 Indirect assumptions
 Maintainability
 E.g., following a coding standard leads to better code
quality, understandability and source code documentation
 Reusability
 E.g., writing and executing unit tests for all code leads to
better design, robust code, and better maintainability
 E.g., product line architecture leads to re-usable software
 Hard metrics for following these practices
 Maintainability
 E.g., coding standards violations found in code reviews
 Reusability
 E.g., number / amount of unit tests, unit test peer
reviews
 E.g., amount of project specific glue code
HELSINKI UNIVERSITY OF TECHNOLOGY 32

33. SoberIT
Software Business and Engineering Institute
Goal – Question – Metric (GQM)
 Goals define the purpose and objective for measurement
 Purpose
 Quality issue
 Object
 Viewpoint
 Questions characterize and refine the measurement goals
 The goal can be achieved when answers to the questions are
available
 Metrics are identified for each question
 Metrics provide the answers to the questions
HELSINKI UNIVERSITY OF TECHNOLOGY 33

34. SoberIT
Software Business and Engineering Institute
Project Planning
HELSINKI UNIVERSITY OF TECHNOLOGY 34

35. SoberIT
Software Business and Engineering Institute
Planning the project with quality goals and practices
 Describe quality goals and major threats
 Make sure that each quality goal is meaningful and described in
the context of this project
 Make sure that all required quality practices are planned
 Effort estimates
 Schedule
 Resourcing
 Plan how to track quality practices and measure achieved quality
 What measures are used?
 Who, how and when?
 How the measures are collected and used?
 Remember: Quality assurance takes effort and calendar time
HELSINKI UNIVERSITY OF TECHNOLOGY 35

36. SoberIT
Software Business and Engineering Institute
Testing and project planning
 Test releases
 Make clear what and when is released for testing
 Plan what is the required quality level of test releases – and how to
ensure / measure it
 Building and delivering the test release takes time
 Do not forget that fixing takes time and effort
 You don’t know how many bugs will be found
 You must plan how many bugs will be found
 Allocate time and resources to test, fix and re-test (verify the fixes)
 More bugs means more fixing and slower testing
 Test schedules are relative
 Development will be late
 Code will be buggy
 Plan how to handle slipping dates and underestimated workload
HELSINKI UNIVERSITY OF TECHNOLOGY 36

37. SoberIT
Software Business and Engineering Institute
Iterative Testing and Deployment
Iteration Heartbeat Testing and deployment
iterations
Release
t
Release to Release to
testing customer
 Testing, stabilizing, and deployment takes time
 Testing requires many iterations
 Found defects has to be fixed
 Fixes has to be re-tested
 Fixing causes more defects…
 If you do more during development the risk is smaller during
testing and deployment
 If all testing is left at the end, the testing phase is hard to
estimate and plan
HELSINKI UNIVERSITY OF TECHNOLOGY 37
 Waterfall process

38. SoberIT
Software Business and Engineering Institute
Project Tracking and Reporting
HELSINKI UNIVERSITY OF TECHNOLOGY 38

39. SoberIT
Software Business and Engineering Institute
Making quality visible in project tracking
 Keep quality goals visible in project meetings
 Track the planned metrics
 Publish the metrics
 React to deviations
 Connect quality metrics in progress tracking
 What does it mean to get a task done?
 Reviews, unit tests, functional tests, regression tests, demo, …
HELSINKI UNIVERSITY OF TECHNOLOGY 39

40. SoberIT
Software Business and Engineering Institute
Making quality visible in project tracking
 Find out the reasons behind the metrics
 small number of found bugs due to ignored testing tasks
 large number of found bugs due to changed specifications -> broken
test cases
Number of
defects
A B
C D
Low QA quality High QA quality
HELSINKI UNIVERSITY OF TECHNOLOGY 40

41. SoberIT
Software Business and Engineering Institute
Test reporting in project management
 As a project manager you need test reports
 Not as bug descriptions, not as plain numbers of test cases
and bugs
 Test reports describe what has been tested and assess the
current level of achieved quality based on that testing
 Test reports should provide information to assess how well
the stated goals have been achieved
 Important content in test reports
 Assessment of the achieved quality
 What was tested, how thoroughly, what was coverage
 What was found, bugs by severity, issues, metrics
 Progress according to the plan
HELSINKI UNIVERSITY OF TECHNOLOGY 41

42. SoberIT
Software Business and Engineering Institute
Reporting with weak metrics:
Testing Dashboard
Modified from: Kaner et al. 2002. Lessons Learned in Software testing.
 If you cannot have direct quality measures, use indirect
 Use of certain quality assurance practices gives indirectly hints of
the likely quality of the final system
 Qualitative assessment and the reasoning behind is much better
than plain numbers
HELSINKI UNIVERSITY OF TECHNOLOGY 42

43. SoberIT
Software Business and Engineering Institute
Test reporting for steering group
 It depends! (Usually higher level of abstraction, focus on specific
issues) 35
30
25 Designed
20 Implemented
15 Tested
10 Needs rework
5
Finished
0
1 3 5 7 9 11 13 15 17 19 21 23
35
30
25
20
Remaining
15
10 Planned
5
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
HELSINKI UNIVERSITY OF TECHNOLOGY

44. SoberIT
Software Business and Engineering Institute
Quality information Managing project
• Review results • To re-scope an
iteration
• Test results, bug
counts • To focus
resources into
• Development right areas
progress
• To make sure
• Metrics of achieving good
different qualities enough quality
(performance,
load, …) • To find out we are
making the right
product
we need to forecast the likely quality of the
final system when it is under development
HELSINKI UNIVERSITY OF TECHNOLOGY

45. SoberIT
Software Business and Engineering Institute
Quality Assurance Through Time Horizons
Iteration Heartbeat
 Heartbeat time horizon Release
 Quality practices are part of each
development task
t
 Iteration time horizon
 Tasks to assure the quality of an
increment
 Release time horizon
 Tasks to assure the quality of the
whole release
Blue = QA practices
HELSINKI UNIVERSITY OF TECHNOLOGY 45

46. SoberIT
Software Business and Engineering Institute
Quality information and time horizons
Release time horizon
i
Release project i i i i i i
Time
 Not connected to any iterations or milestones – hard to track and
manage
 Typical approach to manage
 Informal reviews
 System testing
 Acceptance testing
 Testing non-functional quality characteristics
 Performance
 Usability
 Load and stress testing
 …
HELSINKI UNIVERSITY OF TECHNOLOGY 46

47. SoberIT
Software Business and Engineering Institute
Quality information and time horizons
Iteration time horizon
i
Release project
Iteration i i i i i i i i i i iii
Time
 Quality assurance goals for each iteration
 Activities that provide information by the end of (each) iteration
 Not necessary connected to individual development tasks
 Information available at the end of each iteration, at latest
 Review results
 Unit test and integration test results
 Function test and system test results
 Quality metrics (e.g. performance, bug counts, test reports)
HELSINKI UNIVERSITY OF TECHNOLOGY 47

48. SoberIT
Software Business and Engineering Institute
Quality information and time horizons
Heartbeat time horizon
i
Release project
Iteration
i i
Heartbeat i i i i i i i i i i i i i i i i
Time
 Quality assurance activities as part of each development task
 Information created early and continuously
 Development progress, bug counts, performed tests, …
 Code review results
 Unit tests and results
 Automatic builds and smoke tests
 Function test and integration test results
HELSINKI UNIVERSITY OF TECHNOLOGY 48

49. SoberIT
Software Business and Engineering Institute
Quality information and time horizons
i
Release project i i i i i i
Iteration i i i i i i i i i i iii
i i
Heartbeat i i i i i i i i i i i i i i i i
Time
Heartbeat quality practices Release quality practices
• Information for steering the • Independent activities
iteration • Tracking and planning iterations
• Tracking progress
Iteration quality practices
• Information for steering the current
and forthcoming iterations
• Evaluating achieved quality
HELSINKI UNIVERSITY OF TECHNOLOGY 49

50. SoberIT
Software Business and Engineering Institute
Gather information on short time
horizons
 Progress against iteration (quality) goals must be tracked in
heartbeat rhythm
 Plan how you get the quality information before it’s too late
 Find actively ways of digging quality information out as early and often
as possible
 Talk to testers and developers
 The best way is to get something completed and tested in short
iterations
 The risk of poor quality is not delayed to the end of the project
 Shorter feedback loop -> easier and faster to fix the problems
 Pay attention to heartbeat practices
 Assure good enough quality in each iteration
 Do not push risks like a growing snowball in front of you
HELSINKI UNIVERSITY OF TECHNOLOGY 50

51. SoberIT
Software Business and Engineering Institute
Who is responsible for quality?
 Make sure that quality is everybody’s responsibility
 and it shows in the project plan
 It is good to have a project “QA manager” role
 Responsibility to actively track quality in the heartbeat rhythm and
help project manager to make QA activities to happen
 Reports project’s progress from the quality point of view in the
project meetings
 Tracking quality goals in iteration time horizon
 Tries to be less eager to sacrifice quality for new fancy features
 Keep separate testers tightly in the project
 In the heartbeat rhythm
 Project meetings
 Reviews
 Collaborating with developers
HELSINKI UNIVERSITY OF TECHNOLOGY 51

52. SoberIT
Software Business and Engineering Institute
Summary
 Define concrete and meaningful quality goals
 Identify practices and activities that are applied to achieve the
goals and track progress
 Plan activities and tracking
 quality assurance activities in the schedule and effort estimations
 how to react to deviations
 Remember the different time horizons
 When is the information needed in order to be useful?
 Plan activities to provide the information promptly
 Identify also the major quality threats
 and plan how the threats are handled in the project and when
 Do not leave big threats to be revealed at the end of the project
HELSINKI UNIVERSITY OF TECHNOLOGY 52

53. SoberIT
Software Business and Engineering Institute
Thank you.
Questions?
Tuomas Niinimäki
tuomas.niinimaki@soberit.hut.fi
HELSINKI UNIVERSITY OF TECHNOLOGY