TOC & Kanban in Embedded DomainAn Experience shareRajith R & Girish HegdeHuawei Technologies India Pvt. Ltd. www.huawei.comHUAWEI TECHNOLOGIES CO., LTD.
The SCOPE TOC – Wave1 TOC – Wave2 Project Background Kanban experience sharing • Need for change Learning • Gaps and challenges observed • Strategies to overcome challenges • Benefits seenHUAWEI TECHNOLOGIES CO., LTD. Page 2
Background - Business Line: 2010Huawei India’s Central R & D Business Line predominantly works on Platform Development forTelecom products (e.g. Protocol Stacks, NMS, OS Middleware, Data Middleware, Cloud based solutions, BI System and so on) • 90% teams use agile • Lean Thinking to Improve Efficiency • Time-to-Market still a challenge. Further • VSM to find gaps in process flow Analysis • Wanted efficiency to improve Lead to consider • Deeper analysis of lean practices This is an experience sharing of one of the teams that used TOC and Kanban HUAWEI TECHNOLOGIES CO., LTD. Page 3
Project Background 60+ protocols Maintain over 150+ Product teams 3M LOC 300+ Deployed requirements/per In 500000+ NEs year Network-critical • NE: Network Elements • LOC: Lines of Code We are a Market facing, platform development team and address the demands of various product family teams located worldwide, primarily Shenzhen ChinaHUAWEI TECHNOLOGIES CO., LTD. Page 4
TOC Analysis to find & overcome limiting assumptions (2008) Why Change What to change What to change to How to cause the change System Goal, constraint & Gap Conflicts and current reality Future reality tree Implementation Based on VOC Conflicts association with UDE Break Conflicts with Win-Win How to Change Handling new requirement is not optimized Manage HR better –GAP More HR – better efficiency Deliver frequently using agile Less HR to handle high and faster delivery? load • Shared members, • Manage work in sprints of 3 • Increase availability for active weeks; Use Scrum. components PRS process is • Share members across time consuming More lightweight components in a domain Process? Do it IID way - Follow agile Where we are • UDE: Undesirable Effects • HR: Human Resource HUAWEI TECHNOLOGIES CO., LTD. Page 5
What to change: Conflicts associated with UDEs High Quality Existing procedure Good & PRS process is Fast time consuming Lightweight Fast procedure response Better efficiency Existing HR Efficient & Less HR to handle Fast high load Deliver More HR fasterHUAWEI TECHNOLOGIES CO., LTD. Page 6
What to change to? Break conflicts with Win : WinAssumption Injection How toUDE: PRS process is time consumingCurrent PRS procedures cannot ensure fast Current PRS procedures can ensure fast Do it the IID way. Follow Agile development method.response responseUDE: Less HR to handle high load HR can be managed better to ensure fasterHR should be added to ensure faster delivery Deliver frequently using IID/Agile. delivery HR can be managed better to ensure good Share HR across related domains, across more than oneAdding HR will reduce efficiency efficiency component. Adoption of Iterative Incremental Development was a Radical Change to the team which addressed many of our problems HUAWEI TECHNOLOGIES CO., LTD. Page 7
TOC Analysis to find & overcome further challenges (2010) Why Change What to change What to change to How to cause the change System Goal, Constraint & Gap Conflicts and current reality Future reality tree Implementation Team Challenges Conflicts association with UDE Break Conflicts with Win-Win How to Change Inability to de-scope deliverables • Prioritize work every day and Improve multitasking ability Multitasking is taxing; One finish till end as one piece ofGAP Reduced time for to improve efficiency? Piece Flow is efficient System Testing work, to deliver faster • Use Kanban mode with agile High outburst tasks Extend iteration length to to control queues and delays Control queues and delays accommodate Testing? to get more time • Strengthen visual indicators to see problems early Where we are HUAWEI TECHNOLOGIES CO., LTD. Page 8
What to change to? Break conflicts with Win : WinAssumption Injection How toUDE: Insufficient time for Testing Working independently is not a concern.Independently working members pile up the Have better mechanism to visualize the flow. Ensure study Ensuring continuity in the flow is important towork in the end, reduces testing time flow of value across different states, reduce queues. improve efficiency. Don’t have to reduce testing cycle, if work isRe-plan sprints or reduce testing cycle Operate in One Piece Flow mode to have continuity in flow. processed to DONE without discontinuityUDE: High outburst activities, but unable to de-scope A process based on task flow visualizationNeed to improve Multitasking ability to handle 1. Limit Work In Progress at any point of time, so that and management can be a better alternativemany outburst activities. changing the scope even later in the iteration is possible. to multi-tasking 2. Optimize “in process lead time” by applying Kanban inDe-scoping is not an option because effort De-scoping is possible even later in the and improve efficiency;has already been burnt. iteration. A medium size requirement be delivered in < 7 Cal Days. Adoption of Kanban was one top deed we thought would work for us and found to be the point of highest leverage HUAWEI TECHNOLOGIES CO., LTD. Page 9
Specific approaches taken• Apply Kanban process within the Scrum Measure and Optimize Lead Time Framework Set lead time goal• Adaptations to Scrum User Stories to MMF (Minimum Marketable Measure lead time using Kanban process Feature) Continuous flow preferred over timeboxed Trigger kaizen when actual lead time > goal sprint Retrospectives to Kaizen Find root cause for exceeding lead time goal Focused on the longest queue and the one top improvement. Identify top improvement Relevant metrics measured: Lead time (Small/Medium/Large) Update and systemize new process HUAWEI TECHNOLOGIES CO., LTD. Page 10
Focus areas and strategies • Lean knowledge in team • Study group • Internal coaching • Kanban framework • Limit WIP • Work cells & swim lanes • Optimize lead time • Lead time goal • kaizen • Improve the lean knowledge in the team by forming study group and employing an internal coach • Create the Kanban board by identifying work cells & deciding the Work-In-Progress (WIP) limits • Measure and optimize lead time using kaizen HUAWEI TECHNOLOGIES CO., LTD. Page 11
Kanban Visualization• Restructured Story wall: Defined 6 States• Limiting WIP at each state• Split teams into two 3 lanes to enable One Piece Flow; Lane1 - 2 components, Lane2 – 2 components, Lane3 – Quality Improvement TODO Analysis (2) Development (2) Testing (2) Release (2) Done L1 F34 F33 F32 F31 L2 F43 F42 F41 QI QI2 QI1 • Rule: Tasks can be moved from one state to next only if WIP space is available. No tasks movement if WIP limit has reached • Simple template for Kanban cards to record start date, end date to measure the lead time • Bi-Weekly Kaizen, focus on optimizing lead time HUAWEI TECHNOLOGIES CO., LTD.
Challenges Faced & How we addressed Challenges Initiatives taken • Limiting WIP was tough • Started recording Expected End date (lead time goal) • Sharing team members resulted in some task & time taken at each state to know about big delays. states reaching the WIP limit often • Record backflow items & reasons for delays on the • Lead time measurement & optimizing Kanban cards. • Reducing task switch due to frequent outburst tasks HUAWEI TECHNOLOGIES CO., LTD.
Further improvements Added 1 new swim-lane and reduced WIP limit for some task states • Revisit WIP limits • Improvise swim lanes • Green channel • Measure lead time • Event triggered kaizen • Quality Improvements Green Channel – for handling outburst tasks Event triggered kaizen – Don’t delay improvements TODO Analysis (1) Implementation(1) Review (2) Testing (2) Release (2) Done F37 F32 F35 F31 L1 F34 F36 F33 Green channel - Express Way L2 HUAWEI TECHNOLOGIES CO., LTD. Page 14
Benefits observed WIP limits & one piece flow helped in reducing task switch & waiting Kaizen helped in Lead time optimization & identifying rework queues. Improvements based on this helped in reducing many quality risks (e.g. through Quality Improvement task force, preventive patches) Explicit visual management (e.g. Green Channel) & work-cells helped in handling outburst tasks like patches & urgent requirements better Could restrain from delivering the requirement which will not be used by Product Teams immediately Could make intermediate releases without much delay, which was earlier linked with time boxed iterations HUAWEI TECHNOLOGIES CO., LTD.
Key lessons learnt from TOC, Kanban & Kaizen Highest Leverage Visualization Flow Optimize Methods used: Kanban Methods used: Work Methods used: Measure Method used board, swim lanes, cells, WIP limits lead time , kaizen POOGI green channel Lessons Lessons Lessons Lessons learnt learnt learnt learnt Highest leverage points are Visualization helps in Regulated flow helps you Optimizing lead time helps often counter intuitive making your process to identify the rocks in reducing the time spent in explicit (problems) queues Keep finding areas of highest leverage to continuously improve HUAWEI TECHNOLOGIES CO., LTD. Page 16