Natalya Rukol gave a presentation on defining and measuring quality at the XP days Ukraine conference in Kiev. She discussed why it is important to measure quality to avoid failures, make clear decisions, and continuously improve. Some key things to measure included product quality both externally through customer satisfaction and support tickets, and internally through testing results and bug rates. She emphasized measuring external quality first to ensure business priorities are met. Effective measurement requires preparation, automation, discipline, and using the plan-do-check-act cycle for continuous learning and improvement.

1. Definition of
Quality
Natalya Rukol, Quality Lab.

2. Introduction
About speaker
• 10+ years in IT
• 9 years in quality
management
• PM, TM, CEO
• She is cool!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

3. Introduction
About speech
1. Why to measure
2. What to measure
3. How to measure
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

4. Why measure?
Don’t measure!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

5. Don’t measure!
No visible fail
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

6. Don’t measure!
Meet your fate
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

7. Don’t measure!
Make a real job!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

8. Don’t measure!
Kill a weak team
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

9. Leave it:
Don’t measure??
XP days Ukraine
Kiev
Projects are usually dead
without measurements
11/10/2013
Natalya Rukol
Definition of Qua

10. What to
measure?
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

11. What to measure?
Product quality
Project status
Kiev
11/10/2013
• Control
• React in time
Team efficacy
XP days Ukraine
• Make clear
decisions
• Visualize your
goals
• Better planning
• Improvement
strategy
Natalya Rukol
Definition of Qua

12. What to measure
Product quality
How it
looks
XP days Ukraine
What’s
inside
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

13. Product quality
External quality
• Customers’
satisfaction
• Business results
• Refunds (N & %)
• Support & Hotline
statistics
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

14. Product quality
External quality
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

15. Product external quality
Examples
Metric
6.5 Release
7.0 Release
Customers’ satisfaction
4.4
4.2
Refunds
115
109
Refunds %
0,06%
0,11%
Support time per client, hours
6
8
Support tickets amount per
month
4112
3640
Support tickets in %
(bugs/questions/praises)
27/61/12
19/79/2
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Status
?
Definition of Qua

16. Product external quality
Examples
Metric
6.5 Release
7.0 Release
Customers’ satisfaction
4.4
4.2
Refunds
115
109
Refunds %
0,06%
0,11%
Support time per client, hours
6
8
Support tickets amount per
month
4112
3640
Support tickets in %
(bugs/questions/praises)
27/61/12
19/79/2
Status
?
OK, and what???
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

17. Product external quality
How to use
• Track changes
while it’s not too
late
• Investigate WHY
problems?
• Ask boss for
increase 
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

18. Product external quality
Too late…
Your
software is
shit!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

19. Product quality
Let’s go inside
• Functionality
(what?)
• Non-functional
requirements
(how?)
• Usability
• Performance
• Etc.
• Stability (when?)
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

20. Product quality
Measure inside
What
How to measure
Functionality
(What?)
• Requirements
accepted
• Users scenarios
execution
Non-functional
requirements
(How?)
• Load results
• Performance results
• Usability scenarios pass
ratio
Stability (When?)
• Tests pass rate
• Builds stability
statistics
• Bugs
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

21. Product quality
Internal quality
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

22. Product internal quality
Examples
Metric
7-11/09
14-18/09
20-24/09
Tests pass rate (% OK/Fail/Bugs)
70/12/18
68/13/19
69/12/19
Bugs total / criticals
612 / 28
654 / 19
602 / 24
Bugs average severity
3,7
3,2
3,8
Builds stable / unstable
11 / 0
9/4
6/7
Features accepted
19/21
17/21
20/22
Usability scenarios passed
9/10
n/a
8/10
Load results
18K
18K
6K
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

23. Product internal quality
Not relevant
100% pass rate,
no criticals, blah
blah
Your
software is
still shit!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

24. Product quality
Consequences
What’s
inside
How it
looks
Customers satisfaction
•
•
•
•
Refunds
• Why refunds?
• Collect critics
• Critical bugs
• Features accepted
Support tickets
• What bothers most?
• Usability scenarios
1.
2.
What do they like?
What they use mostly?
What they don’t like?
Pain features?
•
•
•
•
Critical bugs
Testing pass rate
Usability scenarios
Features accepted
You have to find your own consequences!
First find, then prove!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

25. Product quality
Consequences
External quality
Internal quality
XP days Ukraine
Kiev
11/10/2013
• Sales, Refunds
• Satisfaction
• Support cost
• Design
• Code
• Testing
Natalya Rukol
Definition of Qua

26. Product quality
Why external first?
• CLOKs & bugs are
useless w/o
customers &
sales
• No way to forget
priorities
• Think big!
• Tech depends on
business, not vice
versa!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

27. How to
measure?
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

28. How to measure
Do it right!
• Pre-analysis
• Introduction
• Measurement
• Control
• Continuous
improvements
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

29. How to measure
Pre-analysis
• Collaborate with
the whole team
• Remember The
Great Goal (TGG)
• Find your strategy
together
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

30. How to measure
Soft introduction
• Don’t force
• Share
responsibility
• Find enough time
• Provide with
enough resources
• Inside out
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

31. How to measure
Discipline
• Automate all the
measurements
• Show your
respect
• Grow a habit
• Stand it for 2
months
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

32. How to measure
PDCA
• Plan, Do, Check, Act
• Post-Mortems
• Accept mistakes
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

33. Quality Definition:
Results
• Clear view
• Achievements
• Team
collaboration
• Karma increase
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

34. Quality Definition:
What’s next
Contact with any
questions:
Additional
materials:
http://natalyarukol.ru
http://natalyarukol.ru/definitionof-quality
http://nrukol.moikrug.ru
http://software-testing.ru
Skype: natalya.rukol
mailto: natalya@quality-lab.ru
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua