Доклад о внедрении способов оценки программных проектов и программных продуктов.

1. Definition of
Quality
Natalya Rukol, Quality Lab.

2. Introduction
About speaker
• 10+ years in IT
• 9 years in quality
management
• PM, TM, CEO
• She is cool!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

3. Introduction
About speech
1. Why to measure
2. What to measure
3. How to measure
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

4. Why measure?
Don’t measure!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

5. Don’t measure!
No visible fail
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

6. Don’t measure!
Meet your fate
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

7. Don’t measure!
Make a real job!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

8. Don’t measure!
Kill a weak team
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

9. Leave it:
Don’t measure??
XP days Ukraine
Kiev
Projects are usually dead
without measurements
11/10/2013
Natalya Rukol
Definition of Qua

10. What to
measure?
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

11. What to measure?
Product quality
Project status
Kiev
11/10/2013
• Control
• React in time
Team efficacy
XP days Ukraine
• Make clear
decisions
• Visualize your
goals
• Better planning
• Improvement
strategy
Natalya Rukol
Definition of Qua

12. What to measure
Product quality
How it
looks
XP days Ukraine
What’s
inside
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

13. Product quality
External quality
• Customers’
satisfaction
• Business results
• Refunds (N & %)
• Support & Hotline
statistics
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

14. Product quality
External quality
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

15. Product external quality
Examples
Metric
6.5 Release
7.0 Release
Customers’ satisfaction
4.4
4.2
Refunds
115
109
Refunds %
0,06%
0,11%
Support time per client, hours
6
8
Support tickets amount per
month
4112
3640
Support tickets in %
(bugs/questions/praises)
27/61/12
19/79/2
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Status
?
Definition of Qua

16. Product external quality
Examples
Metric
6.5 Release
7.0 Release
Customers’ satisfaction
4.4
4.2
Refunds
115
109
Refunds %
0,06%
0,11%
Support time per client, hours
6
8
Support tickets amount per
month
4112
3640
Support tickets in %
(bugs/questions/praises)
27/61/12
19/79/2
Status
?
OK, and what???
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

17. Product external quality
How to use
• Track changes
while it’s not too
late
• Investigate WHY
problems?
• Ask boss for
increase 
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

18. Product external quality
Too late…
Your
software is
shit!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

19. Product quality
Let’s go inside
• Functionality
(what?)
• Non-functional
requirements
(how?)
• Usability
• Performance
• Etc.
• Stability (when?)
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

20. Product quality
Measure inside
What
How to measure
Functionality
(What?)
• Requirements
accepted
• Users scenarios
execution
Non-functional
requirements
(How?)
• Load results
• Performance results
• Usability scenarios pass
ratio
Stability (When?)
• Tests pass rate
• Builds stability
statistics
• Bugs
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

21. Product quality
Internal quality
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

22. Product internal quality
Examples
Metric
7-11/09
14-18/09
20-24/09
Tests pass rate (% OK/Fail/Bugs)
70/12/18
68/13/19
69/12/19
Bugs total / criticals
612 / 28
654 / 19
602 / 24
Bugs average severity
3,7
3,2
3,8
Builds stable / unstable
11 / 0
9/4
6/7
Features accepted
19/21
17/21
20/22
Usability scenarios passed
9/10
n/a
8/10
Load results
18K
18K
6K
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

23. Product internal quality
Not relevant
100% pass rate,
no criticals, blah
blah
Your
software is
still shit!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

24. Product quality
Consequences
What’s
inside
How it
looks
Customers satisfaction
•
•
•
•
Refunds
• Why refunds?
• Collect critics
• Critical bugs
• Features accepted
Support tickets
• What bothers most?
• Usability scenarios
1.
2.
What do they like?
What they use mostly?
What they don’t like?
Pain features?
•
•
•
•
Critical bugs
Testing pass rate
Usability scenarios
Features accepted
You have to find your own consequences!
First find, then prove!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

25. Product quality
Consequences
External quality
Internal quality
XP days Ukraine
Kiev
11/10/2013
• Sales, Refunds
• Satisfaction
• Support cost
• Design
• Code
• Testing
Natalya Rukol
Definition of Qua

26. Product quality
Why external first?
• CLOKs & bugs are
useless w/o
customers &
sales
• No way to forget
priorities
• Think big!
• Tech depends on
business, not vice
versa!
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

27. How to
measure?
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

28. How to measure
Do it right!
• Pre-analysis
• Introduction
• Measurement
• Control
• Continuous
improvements
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

29. How to measure
Pre-analysis
• Collaborate with
the whole team
• Remember The
Great Goal (TGG)
• Find your strategy
together
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

30. How to measure
Soft introduction
• Don’t force
• Share
responsibility
• Find enough time
• Provide with
enough resources
• Inside out
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

31. How to measure
Discipline
• Automate all the
measurements
• Show your
respect
• Grow a habit
• Stand it for 2
months
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

32. How to measure
PDCA
• Plan, Do, Check, Act
• Post-Mortems
• Accept mistakes
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

33. Quality Definition:
Results
• Clear view
• Achievements
• Team
collaboration
• Karma increase
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua

34. Quality Definition:
What’s next
Contact with any
questions:
Additional
materials:
http://natalyarukol.ru
http://natalyarukol.ru/definitionof-quality
http://nrukol.moikrug.ru
http://software-testing.ru
Skype: natalya.rukol
mailto: natalya@quality-lab.ru
XP days Ukraine
Kiev
11/10/2013
Natalya Rukol
Definition of Qua