Velocity19 Berlin: Swarming, Cynefin… and avoiding the problems of becoming a third-line support team
•Download as PPTX, PDF•
3 likes•488 views
Discussion of Swarming and Cynefin as means to better align devops functions into enterprise support channels.
More Related Content
What's hot
What's hot (20)
Similar to Velocity19 Berlin: Swarming, Cynefin… and avoiding the problems of becoming a third-line support team
Similar to Velocity19 Berlin: Swarming, Cynefin… and avoiding the problems of becoming a third-line support team (20)
More from Jon Stevens-Hall
More from Jon Stevens-Hall (20)
Recently uploaded
Recently uploaded (20)
Velocity19 Berlin: Swarming, Cynefin… and avoiding the problems of becoming a third-line support team
- 1. Swarming, Cynefin… and avoiding the problems of becoming a third-line support team Jon Hall Principal Product Manager, BMC © Copyright 2019 BMC Software, Inc. @jonhall_ Velocity 2019, Berlin
- 2. 2© Copyright 2019 BMC Software, Inc. Enterprise services are complex Digital Services Service Subscribers Tools Processes Services Vulnerabilities3rd Party Providers Compliance Environments
- 3. 3© Copyright 2019 BMC Software, Inc. Enterprise Services are Complex Compliance Environments Private Cloud Mainframe Hybrid Cloud Data Centers Public Cloud Digital Services Service Subscribers 3rd Party Providers Tools Processes Services Vulnerabilities Waterfall Agile Lean SLAs DevOps DevSecOps ITIL End Users Agents IT Ops Security Applications
- 4. “IT organizations that have tried to custom- adjust current tools to meet DevOps practices have a failure rate of 80%” DevOps and the Cost of Downtime: Fortune 1000 Best Practice Metrics Quantified (IDC, 2014) @jonhall_
- 5. • New services and applications suddenly appear • Lost visibility when issues go to developers • Lack of knowledge sharing • New kinds of customer, especially external DevOps challenges Service Management orthodoxies… @jonhall_
- 6. • Scaling customer support • Understanding the context of an issue • Adaptation to life “on call” • What to prioritise? Fixes or features? • How to process alerts, particularly if noisy/low-quality. …but enterprise realities challenge DevOps @jonhall_
- 7. “The enterprise space doesn’t move slowly because they’re stupid, or they hate technology. It’s because they have users” —Luke Kanies, Puppet Founder, Configuration Management Camp 2015, Belgium.
- 8. 2018 State of DevOps Report There is room for Tech Support to bring value to DevOps @jonhall_
- 9. LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 1 SUPPORT Classic “Tiered” Support Structure @jonhall_
- 10. Escalation Escalation Deconstructing the “Tiered” Support Structure LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT LEVEL 1 SUPPORT LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS @jonhall_
- 11. @jonhall_ The Org Chart Problem • Formal Structure • The organisation chart • Informal Structure • Realm of influence between individuals • Value Creation Structure • How work actually gets done
- 12. • Work-in-progress queues • Asynchronous communication • Single role teams • Individual over-exposure • Lack of knowledge sharing How to annoy a DevOps practitioner @jonhall_
- 13. LEVEL 2 SUPPORT LEVEL 2 SUPPORTLEVEL 2 SUPPORT LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 3 SPECIALISTS LEVEL 1 SUPPORT Classic “Tiered” Support Structure @jonhall_ You are here?
- 14. Swarming Network Collaborative Dynamic, loopy Measured by value creation Tiered support Siloes and hierarchies Directed Linear, rigid Measured on activity @jonhall_
- 15. involves removing the tiers of support, and calling on the collective expertise of a “swarm” of analysts. https://www.serviceinnovation.org/intelligent-swarming/ Swarming… @jonhall_
- 16. “I have probably doubled my knowledge of the products in a year because of Swarming… and I have been here a long time” Senior Support Analyst, BMC @jonhall_
- 17. Severity 1 Swarm Prioritise Swarming Process at BMC @jonhall_
- 18. • Rapid responders • Three agents on a scheduled one-week rotation • Primary focus: Provide immediate response, and resolve as soon as possible Swarm lead Communications Other members Research, coordinate, test Severity 1 Swarm @jonhall_
- 19. Severity 1 Swarm Local Dispatch Swarm Prioritise 30% solved here Swarming Process at BMC @jonhall_
- 20. • “Cherry pickers” • Meet every 60-90 minutes • Primary focus: Can new tickets be resolved immediately? • Also: Validation of ticket details before assignment to specialists Experienced analyst Less-experienced analyst Dispatch Swarm @jonhall_
- 21. Local Product-Line Support Teams Severity 1 Swarm Local Dispatch Swarm Prioritise Swarming Process at BMC @jonhall_
- 22. Local Product Line Support Teams Severity 1 Swarm Local Dispatch Swarm Prioritise Severity 1 Swarm Local Dispatch Swarm Prioritise Local Product Line Support Teams Swarming Process at BMC @jonhall_
- 23. Local Product Line Support Teams Local Product Line Support Teams Backlog Swarm Backlog Swarm Backlog Swarm Swarming Process at BMC @jonhall_
- 24. • Global fixers of troublesome tickets • Meet regularly (often several times a day) • Primary focus: Challenging 3rd-line tickets • Replace reassignments and individual assignments Experienced analysts R&D Engineers Backlog Swarms @jonhall_
- 25. • Guidelines, not rules • Metrics had to change (because old ones broke!) • Some people needed help to became more customer facing • Banned ticket tennis and direct escalations to experts • New tooling practices, particularly mobile and chat Things we had to do to make it work @jonhall_
- 26. • 25% median resolution time improvement • Customer satisfaction up 8 points • More issues closed in <2 days • Significant reduction in backlogs • Halved on-boarding time • Freed resources for innovative offerings ~1 year results at BMC @jonhall_
- 27. “Swarming works better. I am able to get multiple experiences from swarm attendees of similar cases they have worked. If there are no experiences, then it’s perspectives: Decades of experience” Senior Support Analyst, BMC @jonhall_
- 28. Swarming at a Global Telco: Expert support for service desk @jonhall_ CUSTOMER CHAT SESSIONS Service Desk Agents CHAT CHANNEL Subject Experts CHAT CHANNEL Subject Experts CHAT CHANNEL Subject Experts • Regional chat-based service desk at a global Telco • Agents can put customer on-hold for 3 minutes • Subject experts wait in “always-on” chat channels
- 29. Swarming example: Auto manufacturer’s connected cars team @jonhall_ Engineering Team A • First responder initiates and coordinates swarms for big issues • Other teams have 1 person on rotation for swarming • Swarms may also involve 3rd parties (e.g. Amazon, Microsoft) • Swarm grows and shrinks as needed Engineering Team B 3rd Party Suppliers First Responder Challenge: how to scale support to millions of new vehicles every year.
- 30. • Perceived increase in “per record” cost • Difficult to evaluate individual contributions • “Cradle to grave” ownership across time zones • Dominant individuals • Finding the right people for a swarm is difficult Issues reported by Swarming early adopters @jonhall_
- 31. Swarming aligns really well to DevOps • Autonomy and self-organisation • Knowledge transfer and skills development • ChatOps, not email • Prevention of accumulation of queued work • Protection of individuals from burnout @jonhall_
- 32. ITSM is accelerating its learning from DevOps ITIL® 4 Foundation (2019) VeriSM – A service management approach for the digital age (2017) ...and it must change in many areas to enable new thinking
- 33. Complex systems fail in complex ways @jonhall_ “10% of nodes enter a simultaneous crash loop cycle, about five times a day, at unpredictable intervals. It clears up before we can debug it” “We run a platform, and it’s hard to distinguish between problems that users are inflicting on themselves, and problems in our own code, since they all manifest as the same errors or timeouts”. “I have 20 microservices and three datastores across three regions, and everything seems to be getting a little slower over the past 2 weeks …but nothing has changed that we know of. Latency is usually back to the historical norm on Tuesdays” So… who do you assign to? Charity Majors Observability for emerging infra Config Management Camp, Ghent 2019
- 34. • Wait… What? • Pronounced “kuh-nev-in” • Developed by Dave Snowden while at IBM in 1999 • “Decision support framework which comes from a mixture of complexity theory and cognitive science… the opposite of a one-size fits all model” Swarming as a means of delivering Cynefin? @jonhall_
- 35. @jonhall_ • Obvious and Complicated domains: • Repeating relationship between cause and effect • With Complicated you need to do analysis to find that relationship • Complex domain: • Understanding the problem requires experimentation and analysis. • May, over time, be able to move to Complicated • Chaotic domain: • Dramatic and unconstrained • Focus on damage limitation, try to move to another domain
- 36. “Obvious” Domain @jonhall_ • “Sense, Categorise, Respond” • Can apply best practice • Template/knowledge-driven resolution • Self service
- 37. “Complicated” Domain @jonhall_ • “Sense, Analyse, Respond” • Good practice. • Dispatch-type swarm – pair up agents with varied experience • Capture detailed knowledge for organizational learning
- 38. Identify “coherent” hypotheses Cynefin approach in a Complex system @jonhall_ • “Sense, Analyse, Respond” • Identify multiple hypotheses • Gain understanding of the system by interacting with it. • Create predictability, increase constraints, hopefully to enable move to Complicated Convene “safe to fail” experiments Observe and monitor impact Amplify good patterns, dampen bad
- 39. “Complex” Domain • “Probe, Sense, Respond” @jonhall_
- 40. “Complex” Domain • “Probe, Sense, Respond” @jonhall_
- 41. “Chaotic” Domain • “Act, Sense, Respond” • Sub-swarms • Deal with the acute situation • Try to discover sufficient information to move to complex @jonhall_
- 42. • Service Management needs to evolve its practices and tooling deliver specific new value to you. • We need to listen to how support is affecting your role, as your impact grows in your enterprise. • You are agents of change in enterprises, with a good opportunity to influence thinking. What next? @jonhall_
- 43. serviceinnovation.org/intelligent-swarming @jonhall_ http://www.bosslevelpodcast.com /dave-snowden-on-complexity- theory-and-astrology/ http://medium.com/@jonhall_ https://youtu.be/fOdtgHu_KeA Consortium for Service Innovation: Intelligent Swarming Boss Level Podcast: Dave Snowden on Cynefin Long-form blog on why Swarming works better for DevOps Charity Majors at #cfgmgmtcamp Observability for emerging infra @mipsytipsy @matthewpskelton @manupaisable @NielsPflaeging
Editor's Notes
- https://www.slideshare.net/JonHall7/is-devops-really-changing-it-support
- Experiments should be parallel – Otherwise, because you’re doing something novel, it is likely to be seen to be successful. Experiments might be naïve Enabling constraints: channel activity, focus it, enable people to do what they wouldn’t normally. Dispositional, not causal. Can make statements about the preset.
- signifies the multiple factors in our environment and our experience that influence us in ways we can never understand
- Experiments should be parallel – Otherwise, because you’re doing something novel, it is likely to be seen to be successful. Experiments might be naïve Enabling constraints: channel activity, focus it, enable people to do what they wouldn’t normally. Dispositional, not causal. Can make statements about the preset.