Skip to main content
Photo by Bert van Dijk - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/75478114@N00 
Created with Haiku Deck
Photo by Timothy Wildey - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/36662615@N05 
Created with Haiku Deck
Photo by -Dreamflow- - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/46070773@N05 
Created with Haiku Deck
Photo by bixentro - Creative Commons Attribution License https://www.flickr.com/photos/82947612@N00 
Created with Haiku Deck
Photo by James Marvin Phelps - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/66727626@N00 
Created with Haiku Deck
Photo by dcmaster - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/76224602@N00 
Created with Haiku Deck
Photo by Great Beyond - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/26104563@N00 
Created with Haiku Deck
Photo by "The Wanderer's Eye Photography" - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/8997331@N04 
Created with Haiku Deck
Photo by -iamok- - Creative Commons Attribution License https://www.flickr.com/photos/8854069@N03 
Created with Haiku Deck
Photo by santiago nicolau - Creative Commons Attribution-ShareAlike License https://www.flickr.com/photos/23552357@N05 
Created with Haiku Deck
Photo by Loving Earth - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/26235754@N05 
Created with Haiku Deck
Photo by dullhunk - Creative Commons Attribution License https://www.flickr.com/photos/14829735@N00 
Created with Haiku Deck
Photo by markhillary - Creative Commons Attribution License https://www.flickr.com/photos/56087830@N00 
Created with Haiku Deck
Photo by orcmid - Creative Commons Attribution License https://www.flickr.com/photos/91555706@N00 
Created with Haiku Deck
Photo by Neil Melville-Kenney - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/27774670@N07 
Created with Haiku Deck
Photo by Brett Whaley - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/21920030@N02 
Created with Haiku Deck
Photo by raneko - Creative Commons Attribution License https://www.flickr.com/photos/24926669@N07 
Created with Haiku Deck
Photo by Photosightfaces - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/30595068@N06 
Created with Haiku Deck
Photo by geezaweezer - Creative Commons Attribution License https://www.flickr.com/photos/33909206@N04 
Created with Haiku Deck
Photo by familymwr - Creative Commons Attribution License https://www.flickr.com/photos/36196762@N04 
Created with Haiku Deck
Photo by Mónica, M - Creative Commons Attribution-NonCommercial License https://www.flickr.com/photos/52329444@N00 
Created with Haiku Deck
Photo by joiseyshowaa - Creative Commons Attribution-ShareAlike License https://www.flickr.com/photos/30201239@N00 
Created with Haiku Deck
Photo by `James Wheeler - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/24128704@N08 
Created with Haiku Deck
Photo by William Christiansen - Creative Commons Attribution License https://www.flickr.com/photos/56222080@N04 
Created with Haiku Deck
Photo by merrycrafts - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/10314331@N08 
Created with Haiku Deck
Photo by susanvg - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/28362388@N00 
Created with Haiku Deck
Photo by monkeyc.net - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/73584213@N00 
Created with Haiku Deck
Photo by juhansonin - Creative Commons Attribution License https://www.flickr.com/photos/38869431@N00 
Created with Haiku Deck
Our Product 
• Network Management domain 
• Windows-based specialized hardware (“Appliances”) 
• Installed in data centers for traffic monitoring, analysis 
and network troubleshooting 
• but not generally on production network 
• Typical users are technical folks – CIO, Network 
Manager, Network Engineers 
• Selling cycles typically align with quarterly or annual 
budget cycles 
• Many sales require implementing customer specials
Old Process, circa 2003 
• Customer Bugs prioritized based on multiple business parameters, 
including (partial list) - 
• Severity 
• Impact on Revenue, Volume, Competitive, etc. 
• Case age 
• PMO would prepare Maintenance Release Plan of Record (MR POR) 
and get buy-in for various types of MRs - 
• Service Packs – bunch up ~50-60 bugs typically every quarter 
• Hot Fixes – 1-2 high-urgency bugs that can’t wait until next SP 
• Patches – workaround for customer-specific issues 
• SPs would have 
• Above The Line (ATL) requirements – must fix 
• Below The Line (BTL) requirements – fix if time permits
Problems with Old Process 
• Dev team had no bandwidth to take on maintenance releases 
• Huge pile of customer escalations without “home” 
• Compounded by high incoming field rate 
• Low closure rate (largely due to no dedicated resources) 
• Large wait for customers to get bug fixes 
• Tech Support often tasked team directly and broke the process 
• Hot fixes not always available to all customers 
• Sometimes, a new bug fix might break a hot fix 
• If a hot fix failed in the field, rollbacks would be very difficult 
• Difficult to estimate time to resolve a bug and give an ETA 
• High-priority bugs could arrive at any time 
• Customer specials could arrive anytime with top priority 
• High internal rejection rate of bug fixes by Tech Support
New Process, 2004-05 
Dedicated 
Customer 
Sustaining 
Team 
New 
“Cumulative 
Hot Fix” 
process 
Improved 
collaboration 
with all 
stakeholders
Our Kanban Process in action… 
GUI 
Product A 
Backend 
Product B 
Product C 
Product D 
Protocols 
Device Drivers 
PMO 
CST 
Manager 
Tech 
Support 
QA Team 
Dev 
Team 
= 
15 
WIP 
= 
15 
WIP 
= 
2 
WIP 
= 
1wk 
WIP 
= 
1wk 
Queue 
= 
0 
Queue 
= 
Infinite
Process improvement…the 
beginnings…
Cumulative Hot Fix Process
Weekly Build Process
So, what is happening? 
• Though not an originally stated vision or goal, the “Work in 
Progress” (WIP) is being limited to # of team members 
• At any time, one developer is assigned only one piece of 
work, thereby achieving “One-Piece Flow” 
• New work is only assigned when current work is 
completed (or cancelled/stalled), and a team member is 
available 
• No wait state or switching costs at an individual level 
• Smaller lead time for bugs (in contract to lead time for SP) 
• The process is allowing ‘continuous deployment’ of each of 
the hot fixes – even though it is only being practiced in a 
limited manner 
• Finally, the flexibility gained is not a zero-sum game – there is 
no penalty on performance in rest of the process
Did this move the needle? 
• Bugs addressed each quarter 
• Quality of bug fixes 
• “Homes” for bugs 
• Total bugs open 
• Open days open 
• People motivation
From SPs to Cumulative Hot Fixes 
Maintenance Releases (Service Packs, Patches, Hot Fixes) 
Q3 2003 Through Q4 2006 (Fiscal Year) 
Hot Fixes 
Patches 
5 2 1 
5 3 1 0 
0 1 0 0 0 
1 
0 2 3 4 4 4 4 2 2 2 1 2 3 1 
0 
2 
8 
11 
16 18 20 
62 
28 27 
26 26 24 
28 
21 
100 
92 
87 
96 
88 
92 
85 
80 
97 96 
93 
100 
94 
91 
7 
12 
15 
25 25 25 
30 30 
28 27 26 
32 
22 
66 
70 
60 
50 
40 
30 
20 
10 
0 
Q3 03 Q4 03 Q1 04 Q2 04 Q3 04 Q4 04 Q1 05 Q2 05 Q3 05 Q4 05 Q1 05 Q2 06 Q3 06 Q4 06 
100 
90 
80 
70 
60 
50 
40 
30 
20 
10 
0 
Service Packs Patches Hot Fixes % Successful 
Percentage of released 
Maintenance Releases 
(Service Packs, 
Patches, Hots Fixes) 
that addressed 
customer reported 
Total Maintenance 
Releases (Service 
Packs, Patches, Hot 
Fixes) for this 
quarter. 
Service 
Packs
Increase in bugs with “homes” 
Customer Direct and Indirect % w/Homes - Historical 
Week Ending February 03, 2006 
55 
Percent with Releases Identified: 56% (Driect), 80% Indirect, 68% (Combined) 
46 45 47 44 45 45 44 
55 
52 54 56 
35 
45 
39 38 41 
51 
46 
31 
40 40 
59 
63 60 61 61 
68 69 
74 72 69 69 
55 
73 
80 
68 
61 
5356 
48 
54 55 
52 51 51 51 50 49 
54 55 
47 47 
11 
15 18 16 
47 
53 55 
43 42 43 
75 
82 
76 76 75 
82 83 82 81 
71 72 
65 
87 
82 
75 
47 
51 51 50 48 48 48 48 48 
53 54 51 51 
21 
27 27 25 
44 
52 51 
41 42 
67 
72 
68 69 68 
75 76 78 76 
70 71 
60 
80 
71 
64 
44 47 
100 
90 
80 
70 
60 
50 
40 
30 
20 
10 
0 
5/6/2005 
5/13/2005 
5/20/2005 
5/27/2005 
6/3/2005 
6/10/2005 
6/17/2005 
6/24/2005 
7/1/2005 
7/8/2005 
8/5/2005 
8/12/2005 
8/19/2005 
8/26/2005 
7/15/2005 
7/22/2005 
7/29/2005 
9/2/2005 
9/9/2005 
10/14/2005 
10/21/2005 
10/28/2005 
9/16/2005 
9/23/2005 
9/30/2005 
10/7/2005 
11/4/2005 
11/11/2005 
11/18/2005 
11/25/2005 
12/16/2005 
12/23/2005 
12/30/2005 
12/2/2005 
12/9/2005 
1/6/2006 
1/13/2006 
1/20/2006 
1/27/2006 
2/3/2006 
Direct Indirect Goal (80% w/Homes) Combined
Total Bugs Open 
Severity Across Products (Historical) 
Week Ending March 31, 2006 
222 
243 
191 
224 
213 
236 
230 
210211 
201199 
196197195 
161 
147146146146 
132 
122 
131 
121 
127 
127125 
111 
148150 
65 
605857 
57 
53 
626466676768 71747575 
6866 
5049495052 
494746 
69707069 
6063 
7473 
68 
6160 
5555 
4749 5149 
44 
3538 
323332343536 
130 
120 
110 
100 
90 
80 
70 
60 
50 
40 
30 
20 
10 
0 
01-Oct-04 
31-Dec-03 
29-Oct-04 
19-Nov-04 
10-Dec-04 
21-Jan-05 
31-Dec-04 
11-Feb-05 
04-Mar-05 
25-Mar-05 
06-May-05 
15-Apr-05 
27-May-05 
17-Jun-05 
08-Jul-05 
29-Jul-05 
19-Aug-05 
09-Sep-05 
21-Oct-05 
30-Sep-05 
11-Nov-05 
02-Dec-05 
13-Jan-06 
23-Dec-05 
03-Feb-06 
24-Feb-06 
260 
240 
220 
200 
180 
160 
140 
120 
100 
80 
60 
40 
20 
0 
17-Mar-06 
Total Severity 1 Severity 2 Severity 3 Severity 4 Severity 5
Average Days Open 
Severity: Average Days By Severity (Historical) 
Week Ending March 31, 2006 
203 
155 
244242246246244248 
136 
234235 
232 
226223 
260 
244 
275275 
268 
299 
232 
210207 
201 
179 
165 
172 
162 
156 
180 
168 
163 
156152 
147 
142 
139143 
132135 
148 
140 
155 
144147146144 
145 
135 
157156156 
152 
148 
149 
144 
140 
133 
154155157 
144147 
177179183182 
163166 
164162165 
154 
146 
149 
141 
134134 
123 
117 
123 
500 
475 
450 
425 
400 
375 
350 
325 
300 
275 
250 
225 
200 
175 
150 
125 
100 
75 
50 
25 
0 
01-Oct-04 
31-Dec-03 
29-Oct-04 
19-Nov-04 
10-Dec-04 
21-Jan-05 
31-Dec-04 
11-Feb-05 
04-Mar-05 
25-Mar-05 
06-May-05 
15-Apr-05 
27-May-05 
17-Jun-05 
08-Jul-05 
29-Jul-05 
19-Aug-05 
09-Sep-05 
21-Oct-05 
30-Sep-05 
11-Nov-05 
02-Dec-05 
13-Jan-06 
23-Dec-05 
03-Feb-06 
24-Feb-06 
300 
275 
250 
225 
200 
175 
150 
125 
100 
75 
50 
25 
0 
17-Mar-06 
Avg Days Open Severity 1 Severity 2 Severity 3 Severity 4 Severity 5
People motivation 
• Started with 16 people dev team 
• We had zero attrition in the team 
• Once the backlog started coming down, engineers were ramped 
off the team to do new features 
• Eventually dismantled the team and rolled-up engineers into dev 
teams when backlog came down to single digits 
16 16 16 16 16 16 16 16 16 16 16 16 
12 12 12 12 12 
10 10 10 
4 4 4 4 4 
6 6 6 
18 
16 
14 
12 
10 
8 
6 
4 
2 
0 
Resource Staffing 
Sustaining Mainline Approved Headcount
Photo by Romain [ apictureourselves.org ] - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/72898690@N00 
Created with Haiku Deck