how to R&D activities - QA mertics by meira diskin from ceragon

1. R&D Maturity
Diskin Meira
1

2. Agenda
Maturity of What?
R&D Project Maturity
R&D Conclusion
Summary
2 Ceragon Networks proprietary and confidential

3. Maturity of What ?
100
90 R&D Maturity
80
70
60
50
40
30
20 R&D Maturity
10
0
-10 Quality (C,H,M)
-20
-30 Coverage Week Open Resolved
-40 21-Sep-08 6 3
-50
-60 28-Sep-08 5 4
-70 05-Oct-08 2 1
-80
-90 12-Oct-08 0 4
-100 26-Oct-08 21 15
02-Nov-08 10 9
80
70 09-Nov-08 15 15
Week 60 34 16-Nov-08 17 13
50 Medium
23-Nov-08 46 32
40 30-Nov-08 6 25
30 07-Dec-08 16 31
Coverage 20
10
36
1 14 5 High
14-Dec-08 33 14
0 3 9 4 7
2 1
2 4
2 21-Dec-08 34 30
70.00% 58.52% 61.00% 28-Dec-08 15 35
IPI
EMS
Algprithm
System
SW
HW
60.00% 48.63% Critical
04-Jan-09 26 13
50.00% Open Coverage 11-Jan-09 28 27
36.79% 35.00% 200 18-Jan-09 20 16
40.00% Int
26.88% 25-Jan-09 24 35
30.00% 138
SWQA 140 144 143 01-Feb-09 14 18
129 133 126
20.00% 100 7 08-Feb-09 19 22
10.00% VoD 15-Feb-09 34 26
39.09 46.89 22-Feb-09 35 34
0.00% 13.12 19.16 26.71
0 1.65 3.5 01-Mar-09 27 24
6 7 08-Mar-09 12 14
1 3 5
Week (Calendar) 7 15-Mar-09 8 10
22-Mar-09 3 3
29-Mar-09 1
3 Ceragon Networks proprietary and confidential

4. R&D Project Maturity
100
90 R&D Maturity
80
70
60
50
40
30
20 R&D Maturity
10
0
-10 Quality
-20
-30 Coverage
-40
-50
-60
-70
-80
-90
-100
Week
Open Coverage
200
138 129 140 133 144 143
126
100
39.09 46.89
13.12 19.16 26.71
0 1.65 3.5
1 3 5
Week (Calendar) 7
4 Ceragon Networks proprietary and confidential

5. R&D Project Maturity
Coverage
80
70.00%
70
61.00%
60 60.00% 58.52%
34 Medium
50
48.63%
40 50.00%
30
High 40.00% 36.79%
20 36 35.00% Int
10 1 14 5 SWQA
9 7 4 30.00% 26.88%
0 3 4 2 1
2 2 VoD
Algprithm
IPI
EMS
System
SW
HW
Critical
20.00%
10.00%
7
0.00%
6 7
5 Ceragon Networks proprietary and confidential

6. R&D Project Maturity
Open/Resolved per Week
6.0(C,H,M)
Week Open Resolved
01-Apr-07 1 1
08-Apr-07 6 4
15-Apr-07 5 4
22-Apr-07 1 1
29-Apr-07 2 2
13-May-07 2 2
20-May-07 1
27-May-07 1 1
03-Jun-07 3 3
10-Jun-07 1 1
24-Jun-07 1
12-Aug-07 15 1
19-Aug-07 1 5
26-Aug-07 3 1
02-Sep-07 19 3
09-Sep-07 6
16-Sep-07 4 6
23-Sep-07 5 1
30-Sep-07 2 2
07-Oct-07 20 13
14-Oct-07 16 10
21-Oct-07 8 14
28-Oct-07 20 14
04-Nov-07 6 7
11-Nov-07 8 5
18-Nov-07 15 3
25-Nov-07 12
02-Dec-07 17 9
09-Dec-07 39 31
16-Dec-07 28 15
23-Dec-07 14 8
30-Dec-07 10 2
06-Jan-08 43 20
13-Jan-08 21 31
20-Jan-08 29 27
27-Jan-08 26 13
03-Feb-08 19 42
10-Feb-08 29 32
17-Feb-08 25 37
24-Feb-08 17 28
02-Mar-08 47 20
09-Mar-08 41 21
16-Mar-08 29 53
23-Mar-08 43 32
30-Mar-08 36 40
06-Apr-08 44 41
13-Apr-08 29 40
20-Apr-08 29 20
27-Apr-08 37 32
04-May-08 29 22
11-May-08 79 74
18-May-08 66 62
25-May-08 49 49
01-Jun-08 30 29
08-Jun-08 22 12
15-Jun-08 31 41
22-Jun-08 24 24
29-Jun-08 27 19
06-Jul-08 22 26
13-Jul-08 12 24
20-Jul-08 16 9
27-Jul-08 9 8
03-Aug-08 7 6
10-Aug-08 12 10 r&d_work_spaceTeamsValidation & VerificationFormal ReleaseAlbatrossIP-10 6.0 & 6.1
17-Aug-08 5 6
24-Aug-08 2 Statistics Working Days IP-10 6.0 & 6.1.xlsx
6 Ceragon Networks proprietary and confidential

7. R&D Conclusions
Open bugs – reason Reject - Bugs
System – incomplete system design
Reopening – a bug that was resolved, and reopened during re-validation..
New – new bug in the implementation.
Hidden – hidden bug due to coverage overcast.
Design – incompatible design with regard to system requirements – should
have been detected at DR.
Aanother bug – chain of bugs – fixing one bug leads to the opening of another.
7 Ceragon Networks proprietary and confidential

8. R&D Conclusions
Bugs detected per Groups
Group Open bugs Critecal & close/Reso Reject Duplicate Deep Monitor
High lve Freeze
x 139 65 85 38 12 33 18
% 46.17% 21.59% 28.24% 27.34% 8.63% 10.96% 5.98%
xx 59 34 43 13 2 9 6
% 19.60% 11.30% 14.29% 22.03% 3.39% 2.99% 1.99%
xxx 85 57 59 15 8 13 11
% 28.23% 18.94% 19.60% 17.65% 9.41% 4.32% 3.65%
xxx 18 5 9 1 6 1
% 5.90% 1.66% 2.99% 0.00% 5.56% 1.99% 0.33%
∑ 301 161 196 66 23 61 36
∑ % 53.48% 61.11% 21.92% 7.64% 20.26% 11.90%
Open Bugs for Each Group Open bugs
Group Critic high ∑
al
x 14 24 40
xx 8 58 102
xxx 1 17 65
xxxx 9 50
xxxxx 1 1
xxxxx 43
8 Ceragon Networks proprietary and confidential

9. R&D Conclusions
Bugs Detected by Tester
Group Open Monitor Reject Duplicate Change
bugs request
Integration zz 89 3 7 8
1
sss 26 5 8 5
oooo 9
SWQA llll 151 4 16 4
2
hjhjh 465 11 29 13
2
hjhjhj 18 2 2
hjhjh 43 1 3
lolol 14 2 2
dfgdfg 2
VoD Yyyyy 41 1 4
2
Il 26 2
hghgh 23 3 5
Al 10 1 1
St 26 4 2
Ale 62 1 1 2
3
Oth 6
9 Ceragon Networks proprietary and confidential

10. R&D Conclusions
Bugs Detected per module
Detected Module vs Severity Graph
8
7 2
6
1
5
1
4
3
1
6
Detected Module vs Severity Graph
3
1 3 1 1
12
2 2 2 4 2
1 1 1
1 2 2 2 2 2
10 2
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
0
A rm
M em
PM
ystem
E
P A
S
E 1
ed
L l - R ote
P tection
ayside
eb
C
R C
R IC
RU
WC
B T
ic
T ffic
B t
CI
E er t
it
oo
L
th ne
3
In
1/T
F G
IS
P
MM
ID
F
F
P
S
W
F
la
L
em
ra
od
8
S
W
ro
oca
4
6 3 - Medium 2 - High 1 - Critical
5
4
3 5
2
1 1
2 2 2 2 2
1 1 1 1 1 1 1 1
0
ACM
Alarm
Modem
PM
tem
IDC
UC
Loopback
Traffic
F A
1
emote
Protection
Boot
E1/T
PG
Sys
Local - R
3 - Medium 2 - High 1 - Critical
10 Ceragon Networks proprietary and confidential

11. R&D Conclusions
Working Days
Group Sub Plan Done
Reg 9 30.5
PVG Modem 15.5 27
ACM adaptive
mode 17 25
PreDistortion 5 11
Miscellaneous 67
Int Bugs 35
Init 2 8
Cfg 3 19
SW
maintenance 2.5 13
Trap 1 4
protection 12.5 16.5
Security 6 11
NTP 1 4.5
Project Group Plan Dane Not calculate ∑ Done Delta
CFM 5 30
Int 143.5 188 188 44.5 bugs 66
setup
SWQA 100 191 87 278 178 SWQA problem 15
X VoD 109 193 193 84 Consecutive
actions 1 5
Int 90.5 138 102 240 149.5 Traffic 3 7
SWQA 94 159 81 240 146 Radio/ACM 14 22
CFM 16 55
XX VoD 133 212 40 252 119 E1/T1 and
Ethernet Only
Int 53.35 88.45 78.5 166.95 113.6 carrier
SWQA 58.25 91.35 12.5 103.85 45.6 regression 0 11
Analyze
XXX VoD 92 108 72 180 88 Vod problems 40
11 Ceragon Networks proprietary and confidential

12. Summary
12 Ceragon Networks proprietary and confidential

13. Summary
13 Ceragon Networks proprietary and confidential

14. Summary
14 Ceragon Networks proprietary and confidential

15. Thanks
15