DevOps Kaizen: Find and Fix What is Really Behind Your Problems

1.
DevOps Kaizen: Finding andFixing What is Really Behind Your Problems © DTO Solutions, Inc. v1.1

2.
@damonedwards Damon Edwards

3.
DevOps Consulting Automation Design Tools

4.
What’s the realdifference between high-performers and low-performers?

5.
It’s the singlemost important skill for any company to master…

6.
The ability toimprove.

7.
a quick noteabout improvement…

8.
The only problemsthat matter are those that prevent the business from reaching its goals.

9.
OpsDev We are inbusiness to give the customer... • What they want • When they want it • At the lowest cost possible

10.
OpsDev DevOps Problems!

11.
Ah-ha! OpsDev Remove Shorten Lead Time Feedback

12.
Why aren’t mostorganizations “good at getting better”?

13.
Improvement already hasa well known recipe: Plan - Do - Study - Act (PDSA) Other variants: PDCA OODA W. Edwards Deming - 1950 © The Deming Institute

14.
Then why hasn’teveryone followed the recipe?

15.
Main culprit: thework isn’t visible

16.
Photo credit: DocSearls on Flickr The “hands on the keyboard” experience

17.
Photo credit: DocSearls on Flickr OpsDev The “hands on the keyboard” experience

18.
Photo credit: DocSearls on Flickr The “hands on the keyboard” experience I can ﬁx it!

19.
Photo credit: DocSearls on Flickr Local optimization often hurts the system

20.
The management experience:the mirage Ah-ha!

21.
The management experience:the reality ? Ah-ha! “Everything takes too long” “Everyone is busy but not much is getting done” “Things break, and then break again and again” “All hands on deck and heroics to get anything done” “Handoffs are painful. Lots of miscommunication and rewok”

22.
? Ah-ha! The management experience:the reality “Everything takes too long” “Everyone is busy but not much is getting done” “Things break, and then break again and again” “All hands on deck and heroics to get anything done” “Lots of miscommunication and rework”

23.
? Ah-ha! Decisions are moreof a (hopeful) educated guess

25.
1. Make thework visible

26.
1. Make thework visible 2. Align the organization

27.
DevOps Kaizen

28.
DevOps Kaizen • “Kaizen”:Japanese word for improvement

29.
DevOps Kaizen • “Kaizen”:Japanese word for improvement • Modern business context:

30.
DevOps Kaizen • “Kaizen”:Japanese word for improvement • Modern business context: • Continuous improvement

31.
DevOps Kaizen • “Kaizen”:Japanese word for improvement • Modern business context: • Continuous improvement • Systematic, scientific-method approach

32.
DevOps Kaizen • “Kaizen”:Japanese word for improvement • Modern business context: • Continuous improvement • Systematic, scientific-method approach • Total engagement of the workforce

33.
DevOps Kaizen • “Kaizen”:Japanese word for improvement • Modern business context: • Continuous improvement • Systematic, scientific-method approach • Total engagement of the workforce • Valuing small changes as much as large changes

34.
DevOps Kaizen • “Kaizen”:Japanese word for improvement • Modern business context: • Continuous improvement • Systematic, scientific-method approach • Total engagement of the workforce • Valuing small changes as much as large changes • Kaizen in DevOps context:

35.
DevOps Kaizen • “Kaizen”:Japanese word for improvement • Modern business context: • Continuous improvement • Systematic, scientific-method approach • Total engagement of the workforce • Valuing small changes as much as large changes • Kaizen in DevOps context: • Continuously improve the flow of work through the full value stream in order to improve customer outcomes

36.
Service Delivery Metrics Kaizen Program Oversight Planning & Retrospectives Informs Informs Countermeasures & Blockers Elementsof a DevOps Kaizen Program

37.

38.

39.
1. The willto make change happen 2. The resources to make change happen 3. Follow-through / clear obstacles Leading the Improvement

40.
1. The willto make change happen 2. The resources to make change happen 3. Follow-through / clear obstacles Leading the Improvement This (and only this) is what the Kaizen Program Oversight Group does!

41.

42.
Service Delivery Metrics Kaizen Program Oversight Planning & Retrospectives Informs Informs Countermeasures & Blockers Elementsof a DevOps Kaizen Program This is where the work becomes visible!

43.
Map end-to-end process1 Includekey process metrics: Lead Time Processing Time Scrap Rate Head Count DevOps Kaizen: Retrospective Technique

44.
Identify wastes, inefficiencies,bottlenecks PD - Partially Done TS - Task Switching W - Waiting M - Motion / Manual D - Defects EP - Extra Process EF - Extra Features HB - Heroics Structured approach building on DevOps adaptation of “7 deadly wastes” from Lean / Agile: 2 DevOps Kaizen: Retrospective Technique

45.
Creative Business Release Engineering Development QA Technical Operations ProgramManagement L/T Lead time P/T Process time H/C Head count S/R Scrap rate D Defects EP Extra processes M Motion PD Partially done TS Task switching W Waiting Core business service delivery process Current state value stream map Development Environments Release Build and deploy QA Environments Performance Environment Staging Environment Production Environment Merge project branch Deploy to Performance Web Assets, flows and wireframes Delta features and fixesSet of service branches Documented test plans and cases Request QA build Daily Scrum meeting Daily Scrum meeting Deployment Meeting Business Systems Support Stan Walters Change Control Project Manager Erica Switzer Change request Business Analyst Erica Switzer Steering Committee Erica Switzer Full Projects / Expedites Estimates and requirements Technical Specification Deployment Document Change Control Forms RT tickets Jira issues Jira issues Tested packages Development Joe Harbaugh L/T 8 weeks P/T H/C S/R 4 weeks 5 25% Develop web assets Bob McNulty L/T 6 weeks P/T H/C S/R 4 weeks 12 5% Development Build and deploy Joe Harbaugh L/T 2 days P/T H/C S/R 4 hours 1 10% Development Integration testing Joe Harbaugh L/T 1 day P/T H/C S/R 1 day 5 95% Performance testing Deploy to Production Build test plan Raj Lee L/T 5 days P/T H/C S/R 4 days 1 10% Build test cases Raj Lee L/T 2 weeks P/T H/C S/R 8 days 2 20% Fernando Gomez L/T 8 hours P/T H/C S/R 7 hours 1 2% Acceptance testing Raj Lee L/T 5 days P/T H/C S/R 5 hours 2 10% Feature testing Raj Lee L/T 4 weeks P/T H/C S/R 2.5 weeks 4 20% Allen Cannata L/T 1 week P/T H/C S/R 3 days 1 66% RT tickets Cut project branch Chris Holmes L/T 15 minutes P/T H/C S/R 5 minutes 1 0% Deploy to Staging Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Staging testing Frances Middleton L/T 1 week P/T H/C S/R 3.5 days 4 66% Allen Cannata L/T 9 hours P/T H/C S/R 8 hours 6 66% Frances Middleton L/T 6 days P/T H/C S/R 90 minutes 3 66% Fernando Gomez L/T 30 minutes P/T H/C S/R 30 minutes 1 33% D EP Dx2 EP Mx2 W D PD Dx3 Mx3 TSx5 Dx2 Wx2 PDx4 EP D PDx3 Mx8 D Wx2 PDx2 What should you be looking for? • Long lead times (vs processing time) • Request and approval queues • Large batch sizes • Human to human information transfer • Scrap / Rework / “Backwash” • Process or tooling inconsistency • Manual deployment / configuration • Reliance on manual testing / verification • Injecting requirements late in the process • Fixing in place (vs rebuilding) • “Snowflake” servers and environments • Manual dependency management • Ad-hoc or unstructured artifact management

46.
DevOps Kaizen: RetrospectiveTechnique Identify countermeasures Countermeasures must be actionable, backlog ready. Focus on short-term “baby steps”. Note broader, strategic recommendations. 3

47.
DevOps Kaizen: Behaviorsto Encourage

48.
Learn Fast, FailEarly Verification tests catch problems early. Tests accumulate to build the “business immune system. DevOps Kaizen: Behaviors to Encourage

49.
Learn Fast, FailEarly Verification tests catch problems early. Tests accumulate to build the “business immune system. Work in Small Batches Work is planned and executed in small batches (made continuously and operations-ready early). DevOps Kaizen: Behaviors to Encourage

50.
Learn Fast, FailEarly Verification tests catch problems early. Tests accumulate to build the “business immune system. Standard Processes Making Standard Parts Specs and recipes produce infrastructure and app components. No more “snowflakes”. Work in Small Batches Work is planned and executed in small batches (made continuously and operations-ready early). DevOps Kaizen: Behaviors to Encourage

51.
Learn Fast, FailEarly Verification tests catch problems early. Tests accumulate to build the “business immune system. Shared Ops Visibility Live visibility into apps/environments. Single page, simple shared conventions part of DoD Standard Processes Making Standard Parts Specs and recipes produce infrastructure and app components. No more “snowflakes”. Work in Small Batches Work is planned and executed in small batches (made continuously and operations-ready early). DevOps Kaizen: Behaviors to Encourage

52.
Learn Fast, FailEarly Verification tests catch problems early. Tests accumulate to build the “business immune system. Shared Ops Visibility Live visibility into apps/environments. Single page, simple shared conventions part of DoD Standard Processes Making Standard Parts Specs and recipes produce infrastructure and app components. No more “snowflakes”. Ops Requirements are Product Features Performance, Stability, Configurability are all essential features for a successful product. Work in Small Batches Work is planned and executed in small batches (made continuously and operations-ready early). DevOps Kaizen: Behaviors to Encourage

53.
Learn Fast, FailEarly Verification tests catch problems early. Tests accumulate to build the “business immune system. Shared Ops Visibility Live visibility into apps/environments. Single page, simple shared conventions part of DoD Standard Processes Making Standard Parts Specs and recipes produce infrastructure and app components. No more “snowflakes”. Ops Requirements are Product Features Performance, Stability, Configurability are all essential features for a successful product. Self-Service Operations People consume operations services rather than opening tickets for others to do it for them. Work in Small Batches Work is planned and executed in small batches (made continuously and operations-ready early). DevOps Kaizen: Behaviors to Encourage

54.
DevOps Kaizen: RetrospectiveTechnique Create Improvement Storyboards (Kata Style)4

55.
DevOps Kaizen: PlanningTechnique Draw map in this direction What are we going to do? Who do we need to do it? What do we need to do it? What’s going to get in our way?

56.
Parts Fulfillment Networks(a division of Parts Unlimited)

57.
Parts Fulfillment Networks(a division of Parts Unlimited) ● “How do we protect the previous $2B in revenue while innovating towards next $2B… without doubling the team?”

58.
Parts Fulfillment Networks(a division of Parts Unlimited) ● “How do we protect the previous $2B in revenue while innovating towards next $2B… without doubling the team?” ● Schedule slippage and cost of delay impacting customer relationships

59.
Parts Fulfillment Networks(a division of Parts Unlimited) ● “How do we protect the previous $2B in revenue while innovating towards next $2B… without doubling the team?” ● Schedule slippage and cost of delay impacting customer relationships ● Everybody looks busy (40+ concurrent projects) but nothing gets done

60.
Parts Fulfillment Networks(a division of Parts Unlimited) ● “How do we protect the previous $2B in revenue while innovating towards next $2B… without doubling the team?” ● Schedule slippage and cost of delay impacting customer relationships ● Everybody looks busy (40+ concurrent projects) but nothing gets done ● “We love doing things so much we do them 3 or 4 times”

61.
Parts Fulfillment Networks(a division of Parts Unlimited) ● “How do we protect the previous $2B in revenue while innovating towards next $2B… without doubling the team?” ● Schedule slippage and cost of delay impacting customer relationships ● Everybody looks busy (40+ concurrent projects) but nothing gets done ● “We love doing things so much we do them 3 or 4 times” ● “Customers find the problems before we do”

62.
Parts Fulfillment Networks(a division of Parts Unlimited) ● “How do we protect the previous $2B in revenue while innovating towards next $2B… without doubling the team?” ● Schedule slippage and cost of delay impacting customer relationships ● Everybody looks busy (40+ concurrent projects) but nothing gets done ● “We love doing things so much we do them 3 or 4 times” ● “Customers find the problems before we do” ● Business unit leaders pressuring to “route around” Core IT

63.
Punchdown Approvals Parts Unlimited -High Availability for Acme Co. Evaluate Needs Pro Serve Leads Share Point 3 months "Susan T." VP of Vertical Act Mgmt "Pat" Owner of Acme Co Act. Acme Co. Reqs Cust. Config Pro Services PM Begin Contract Negotiation Evaluate Risk for Deliveries to SLA Pro Services Legal Ops Reqs "David" Service Level Manager Finance Reporting Project Delivery ExpectationBU Owner 5 months total Pro Services BA's Write Specs for Fulfillment Share Point .doc Discuss Planning Attendees Project Kickoff Meeting Pro Services Ops PM App Support Share Point App Support Tickets High Level About Project Estimate and Priorities "Linda" Ops PM Child ticket "Project Estimate" Create Project Estimate Package Systems Networks Security Leads Core Network Provider Share Point .ppt BOM. xls Function Specific Project Estimate estimate "BOM.xls" Set Timeline Ops PM Project Estimate Judge Costs & Approval App Support Partner BU Business Case 3 weeks Create App Spec Share Point Tech Leads Review App Spec Dev Leads Pro Services Pro Services PM App Spec.doc Development App Spec Reqs Status Updates Availability Pro Services Lead Acme Co. Share Point Create Build Request Project Estimate "Linda" Ops PM "Linda" "Denis" Ops PM Add Design .xl s Ops Engineer "Chuck" Ops Specialists 1 month Share Point Build. xls Create Change Tickets > 100 Build. xls Ticket System Net Facility Cabling Storage "Linda" Ops PM RESET DELIVERY DATE! Acme Co. Fix Tickets! Senior Ops Engr Create DB Schema Ops DBA Conflict Resolution 3 weeks BA App Dev Net Engr Assign Ticket Ticket System Network Ops "Chris" Ops Engineer Procure Hardware Procurement SaaS Install Hardware SA Install OS Version (2) Jumpstart Images Ticket System Run Cabling Ops Stock Clerk Receive and Tag Create DB VMs (6) Install Hw Req Cluster Setup Creat App servers Solaris Zones (10) Websphere NFS Write Data Migration Scripts App DBA UAT US UAT Europe PROD US PROD Europe F5 Rules Network Ops Pro Services External DNS SSL Certificate Create VIP Create External IPCreate Subnet Firewall Rules Status Event Cust. Config POC Ticket System Create UAT Deploy Ticket Middleware (code) DBA (SQL) Network Approvals "Marvin" Middleware Ops Deploy EAR Retest App Configure to the data sources Notify recipient in Ticket System Admin 01 NFS UAT Cust. ConfigPOC App Support Pro Services Build Server EAR copied (FTP) Promote App Dev Smoke Test App Support Share Point Test Script Run Functional Test Test Plan .xls or .doc Pro Services BAWrite Test Plan Customer Sign Off on Test Plan Acme Co. "Linda" PM Coordinate Operational Readiness Testing UAT PROD Acme Co. Testing Repeated in Prod Middleware DB Migration Re- Sync DB DBA (happened earlier) 1 Month Create Production Readiness Meeting DR Helpdesk Team Leads Apps Ops PM Share Point Ticket System Go Live Ops PM Phone Bridge 1 Week 2 Weeks Turn off old site Run final DB sync Customer uses new VIP Renegotiate Create Build Out Doc QA Test DIT US 4 hour windows W UAT Deploy and QA Go Live .xls Go Live .xls UAT and PROD Network Config UAT and PROD Server Provisioning "Missing Routes" PROD Deploy and QA Create Ops Tickets App Dev and QA App Specification Financial Approval and Project Commitment Project Estimate Sales and Customer Analysis Sales and Delivery Risk Analysis Data Setup PD M W PD D M PD PD M TS D M TS HB HB M TS D PD W M M M M M PD D M M M PD D M TS D W TS D D PD W TS W Ops Requirements VerificationOps Requirements Verification Small Batches Ops Requirements Verification Verification Ops Requirements Standard Parts Small Batches Verification Small Batches Ops Requirements Verification Verification Standard Parts Verification Standard Parts Standard Parts Ops Requirements Verification Ops Requirements Ops Health Dashboard Verification Standard Parts Ops Health Dashboard Ops Health Dashboard Ops Health Dashboard Ops Health Dashboard Standard Parts Verification Verification Verification Ops Requirements Ops Requirements Ops Health Dashboard Verification Standard Parts Standard Parts Reqs Cust. Config App Spec Project EstimateProject Estimate Biz Case Approval Build Doc Design & Spec Project Plan Tickets ~8 months before implementation starts (and information / requirements still incorrect or incomplete!) Tickets Tickets Tickets Ad-Hoc Emails -Missing details -New technical requirements -Missing context Information Flow

64.

65.
Parts Unlimited -High Availability for Acme Co. Evaluate Needs Pro Serve Leads Share Point 3 months "Susan T." VP of Vertical Act Mgmt "Pat" Owner of Acme Co Act. Acme Co. Reqs Cust. Config Pro Services PM Begin Contract Negotiation Evaluate Risk for Deliveries to SLA Pro Services Legal Ops Reqs "David" Service Level Manager Finance Reporting Project Delivery ExpectationBU Owner 5 months total Pro Services BA's Write Specs for Fulfillment Share Point .doc Discuss Planning Attendees Project Kickoff Meeting Pro Services Ops PM App Support Share Point App Support Tickets High Level About Project Estimate and Priorities "Linda" Ops PM Child ticket "Project Estimate" Create Project Estimate Package Systems Networks Security Leads Core Network Provider Share Point .ppt BOM. xls Function Specific Project Estimate estimate "BOM.xls" Set Timeline Ops PM Project Estimate Judge Costs & Approval App Support Partner BU Business Case 3 weeks Create App Spec Share Point Tech Leads Review App Spec Dev Leads Pro Services Pro Services PM App Spec.doc Development App Spec Reqs Status Updates Availability Pro Services Lead Acme Co. Share Point Create Build Request Project Estimate "Linda" Ops PM "Linda" "Denis" Ops PM Add Design .xl s Ops Engineer "Chuck" Ops Specialists 1 month Share Point Build. xls Create Change Tickets > 100 Build. xls Ticket System Net Facility Cabling Storage "Linda" Ops PM RESET DELIVERY DATE! Acme Co. Fix Tickets! Senior Ops Engr Create DB Schema Ops DBA Conflict Resolution 3 weeks BA App Dev Net Engr Assign Ticket Ticket System Network Ops "Chris" Ops Engineer Procure Hardware Procurement SaaS Install Hardware SA Install OS Version (2) Jumpstart Images Ticket System Run Cabling Ops Stock Clerk Receive and Tag Create VMs (6) Install Hw Req Cluster Setup Creat App servers Solaris Zones (10) Websphere UAT US UAT Europe PR Extern DNS Create Subnet Firewall Rules Renegotiate Create Build Out Doc QA Test DIT US UAT and PROD UAT and PROD Server P "Missing Routes" Create Ops Tickets App Dev and QA App Specification Financial Approval and Project Commitment Project Estimate Sales and Customer Analysis Sales and Delivery Risk Analysis Data Setup PD M W PD D M PD PD M TS D M TS HB HB M M M PD D M TS PD Ops Requirements VerificationOps Requirements Verification Small Batches Ops Requirements Verification Verification Ops Requirements Standard Parts Small Batches Verification Small Batches Ops Requirements Verification Verification Standard Parts Verification Standard Parts Stand Ops Health Dashboard Ops He Dashbo Ops Health Dashboard Stand Standard Parts Reqs Cust. Config App Spec Project EstimateProject Estimate Biz Case Approval Build Doc Design & Spec Project Plan Tickets ~8 months before implementation starts (and information / requirements still incorrect or incomplete!) Tickets Information Flow

66.
Parts Unlimited -High Availability for Acme Co. Evaluate Needs Pro Serve Leads Share Point 3 months "Susan T." VP of Vertical Act Mgmt "Pat" Owner of Acme Co Act. Acme Co. Reqs Cust. Config Pro Services PM Begin Contract Negotiation Evaluate Risk for Deliveries to SLA Pro Services Legal Ops Reqs "David" Service Level Manager Finance Reporting Project Delivery ExpectationBU Owner 5 months total Pro Services BA's Write Specs for Fulfillment Share Point .doc Discuss Planning Attendees Project Kickoff Meeting Pro Services Ops PM App Support Share Point App Support Tickets High Level About Project Estimate and Priorities "Linda" Ops PM Child ticket "Project Estimate" Create Project Estimate Package Systems Networks Security Leads Core Network Provider Share Point .ppt BOM. xls Function Specific Project Estimate estimate "BOM.xls" Set Timeline Ops PM Project Estimate Judge Costs & Approval App Support Partner BU Business Case 3 weeks Create App Spec Share Point Tech Leads Review App Spec Dev Leads Pro Services Pro Services PM App Spec.doc Development App Spec Reqs Status Updates Availability Pro Services Lead Acme Co. Share Point Create Build Request Project Estimate "Linda" Ops PM "Linda" "Denis" Ops PM Add Design .xl s Ops Engineer "Chuck" Ops Specialists 1 month Share Point Build. xls Create Change Tickets > 100 Build. xls Ticket System Net Facility Cabling Storage "Linda" Ops PM RESET DELIVERY DATE! Acme Co. Fix Tickets! Senior Ops Engr Create DB Schema Ops DBA Conflict Resolution 3 weeks BA App Dev Net Engr Assign Ticket Ticket System Network Ops "Chris" Ops Engineer Procure Hardware Procurement SaaS Install Hardware SA Install OS Version (2) Jumpstart Images Ticket System Run Cabling Ops Stock Clerk Receive and Tag Create VMs (6) Install Hw Req Cluster Setup Creat App servers Solaris Zones (10) Websphere UAT US UAT Europe PR Extern DNS Create Subnet Firewall Rules Renegotiate Create Build Out Doc QA Test DIT US UAT and PROD UAT and PROD Server P "Missing Routes" Create Ops Tickets App Dev and QA App Specification Financial Approval and Project Commitment Project Estimate Sales and Customer Analysis Sales and Delivery Risk Analysis Data Setup PD M W PD D M PD PD M TS D M TS HB HB M M M PD D M TS PD Ops Requirements VerificationOps Requirements Verification Small Batches Ops Requirements Verification Verification Ops Requirements Standard Parts Small Batches Verification Small Batches Ops Requirements Verification Verification Standard Parts Verification Standard Parts Stand Ops Health Dashboard Ops He Dashbo Ops Health Dashboard Stand Standard Parts Reqs Cust. Config App Spec Project EstimateProject Estimate Biz Case Approval Build Doc Design & Spec Project Plan Tickets ~8 months before implementation starts (and information / requirements still incorrect or incomplete!) Tickets Information Flow + Work in small batches + Early Ops Involvement + Standardized Catalog + Plan and Design by Those Who Do

67.

68.

69.
eate oject mate kage Systems Networks Security Leads Core Network Provider .ppt BOM. xls Function Specific Judge Costs & Approval App Support PartnerBU Business Case 3 weeks Create App Spec Share Point Tech Leads Review App Spec Dev Leads Pro Services Pro Services PM App Spec.doc Development App Spec Reqs Status Updates Availability Pro Services Lead Acme Co. Share Point Create Build Request Project Estimate "Linda" Ops PM "Linda" "Denis" Ops PM Add Design .xl s Ops Engineer "Chuck" Ops Specialists 1 month Share Point Build. xls Create Change Tickets > 100 Build. xls Ticket System Net Facility Cabling Storage "Linda" Ops PM RESET DELIVERY DATE! Acme Co. Fix Tickets! Senior Ops Engr Create DB Schema Ops DBA Conflict Resolution 3 weeks BA App Dev Net Engr Assign Ticket Ticket System Network Ops "Chris" Ops Engineer Procure Hardware Procurement SaaS Install Hardware SA Install OS Version (2) Jumpstart Images Ticket System Run Cabling Ops Stock Clerk Receive and Tag Create DB VMs (6) Install Hw Req Cluster Setup Creat App servers Solaris Zones (10) Websphere NFS Write Data Migration Scripts App DBA UAT US UAT Europe PROD US PROD Europe F5 Rules Network Ops Pro Services External DNS SSL Certificate Create VIP Create External IPCreate Subnet Firewall Rules Status Event Cust. Config POC Ticket System Create UAT Deploy Ticket Middleware (code) DBA (SQL) Network Approvals "Marvin" Middleware Ops Deploy EAR Retest App Configure to the data sources Notify recipient in Ticket System Create Build Out Doc QA Test DIT US UAT and PROD Network Config UAT and PROD Server Provisioning "Missing Routes" Create Ops Tickets App Dev and QA App Specification val tment imate Data Setup D PD M TS D M TS HB HB M TS D PD W M M M M M PD D M TS D D PD TS W Small Batches Ops Requirements Verification Verification Ops Requirements Standard Parts Small Batches Verification Small Batches Ops Requirements Verification Verification Standard Parts Verification Standard Parts Standard Parts Ops Requirements Verification Ops Requirements Ops Health Dashboard Verific Standard Parts Ops Hea Dashbo Ops Health Dashboard Ops Health Dashboard Ops Health Dashboard Standard Parts Standard Parts ~8 months before implementation starts (and information /

70.
eate oject mate kage Systems Networks Security Leads Core Network Provider .ppt BOM. xls Function Specific Judge Costs & Approval App Support PartnerBU Business Case 3 weeks Create App Spec Share Point Tech Leads Review App Spec Dev Leads Pro Services Pro Services PM App Spec.doc Development App Spec Reqs Status Updates Availability Pro Services Lead Acme Co. Share Point Create Build Request Project Estimate "Linda" Ops PM "Linda" "Denis" Ops PM Add Design .xl s Ops Engineer "Chuck" Ops Specialists 1 month Share Point Build. xls Create Change Tickets > 100 Build. xls Ticket System Net Facility Cabling Storage "Linda" Ops PM RESET DELIVERY DATE! Acme Co. Fix Tickets! Senior Ops Engr Create DB Schema Ops DBA Conflict Resolution 3 weeks BA App Dev Net Engr Assign Ticket Ticket System Network Ops "Chris" Ops Engineer Procure Hardware Procurement SaaS Install Hardware SA Install OS Version (2) Jumpstart Images Ticket System Run Cabling Ops Stock Clerk Receive and Tag Create DB VMs (6) Install Hw Req Cluster Setup Creat App servers Solaris Zones (10) Websphere NFS Write Data Migration Scripts App DBA UAT US UAT Europe PROD US PROD Europe F5 Rules Network Ops Pro Services External DNS SSL Certificate Create VIP Create External IPCreate Subnet Firewall Rules Status Event Cust. Config POC Ticket System Create UAT Deploy Ticket Middleware (code) DBA (SQL) Network Approvals "Marvin" Middleware Ops Deploy EAR Retest App Configure to the data sources Notify recipient in Ticket System Create Build Out Doc QA Test DIT US UAT and PROD Network Config UAT and PROD Server Provisioning "Missing Routes" Create Ops Tickets App Dev and QA App Specification val tment imate Data Setup D PD M TS D M TS HB HB M TS D PD W M M M M M PD D M TS D D PD TS W Small Batches Ops Requirements Verification Verification Ops Requirements Standard Parts Small Batches Verification Small Batches Ops Requirements Verification Verification Standard Parts Verification Standard Parts Standard Parts Ops Requirements Verification Ops Requirements Ops Health Dashboard Verific Standard Parts Ops Hea Dashbo Ops Health Dashboard Ops Health Dashboard Ops Health Dashboard Standard Parts Standard Parts ~8 months before implementation starts (and information / + GTM/LTM in Early Environments

71.

72.

73.
un Cabling Receive and Tag CreateDB VMs (6) Install Hw Req Creat App servers Solaris Zones (10) Websphere NFS Write Data Migration Scripts App DBA F5 Rules Network Ops Pro Services External DNS SSL Certificate Create VIP Create External IPCreate Subnet Firewall Rules Status Event Cust. Config POC Ticket System Create UAT Deploy Ticket Middleware (code) DBA (SQL) Network Approvals "Marvin" Middleware Ops Deploy EAR Retest App Configure to the data sources Notify recipient in Ticket System Admin 01 NFS UAT Cust. ConfigPOC App Support Pro Services Build Server EAR copied (FTP) Promote App Dev Smoke Test App Support Share Point Test Script Run Functional Test Test Plan .xls or .doc Pro Services BAWrite Test Plan "Linda" PM Coordinate Operational Readiness Testing UAT PROD Acme Co. Testing Repeated in Prod Middleware DB Migration Re- Sync DB DBA 1 Month Create Production Readiness Meeting DR He Te Ap O 4 hour windows W UAT Deploy and QA UAT and PROD Network Config "Missing Routes" PROD Deploy and QA HB M TS D PD W M M M M PD D M M D W TS D D W TS W Verification n Standard Parts Standard Parts Ops Requirements Verification Ops Requirements Verification Standard Parts Ops Health Dashboard Ops Health Dashboard Standard Parts Verification Verification Verification Ops Requirements Ops Health Dashboard Verification Standard Parts

74.
un Cabling Receive and Tag CreateDB VMs (6) Install Hw Req Creat App servers Solaris Zones (10) Websphere NFS Write Data Migration Scripts App DBA F5 Rules Network Ops Pro Services External DNS SSL Certificate Create VIP Create External IPCreate Subnet Firewall Rules Status Event Cust. Config POC Ticket System Create UAT Deploy Ticket Middleware (code) DBA (SQL) Network Approvals "Marvin" Middleware Ops Deploy EAR Retest App Configure to the data sources Notify recipient in Ticket System Admin 01 NFS UAT Cust. ConfigPOC App Support Pro Services Build Server EAR copied (FTP) Promote App Dev Smoke Test App Support Share Point Test Script Run Functional Test Test Plan .xls or .doc Pro Services BAWrite Test Plan "Linda" PM Coordinate Operational Readiness Testing UAT PROD Acme Co. Testing Repeated in Prod Middleware DB Migration Re- Sync DB DBA 1 Month Create Production Readiness Meeting DR He Te Ap O 4 hour windows W UAT Deploy and QA UAT and PROD Network Config "Missing Routes" PROD Deploy and QA HB M TS D PD W M M M M PD D M M D W TS D D W TS W Verification n Standard Parts Standard Parts Ops Requirements Verification Ops Requirements Verification Standard Parts Ops Health Dashboard Ops Health Dashboard Standard Parts Verification Verification Verification Ops Requirements Ops Health Dashboard Verification Standard Parts + Dev provide verification script

75.

76.
DevOps Kaizen: Designedfor the Enterprise Needs

77.
DevOps Kaizen: Designedfor the Enterprise Needs •Scale improvement quickly

78.
DevOps Kaizen: Designedfor the Enterprise Needs •Scale improvement quickly •Span multiple organizational boundaries

79.
DevOps Kaizen: Designedfor the Enterprise Needs •Scale improvement quickly •Span multiple organizational boundaries •Work with substantial numbers of legacy technologies

80.
DevOps Kaizen: Designedfor the Enterprise Needs •Scale improvement quickly •Span multiple organizational boundaries •Work with substantial numbers of legacy technologies •Develop your existing staﬀ in mass

81.
DevOps Kaizen: Designedfor the Enterprise Needs •Scale improvement quickly •Span multiple organizational boundaries •Work with substantial numbers of legacy technologies •Develop your existing staﬀ in mass •Be self-funding after initial seed investment

82.
“No need forthis. We’ve got a big project lined up that will replace everything and solve our problems…”

83.
The “Big Bang”Transformation Dream Start Finish

84.
The “Big Bang”Transformation Reality Start Finish Fear Panic Abort Maybe

85.
The “Big Bang”Transformation Reality Start Finish Fear Panic Abort Maybe People revert to legacy behaviors

86.
“Big J” vs“Little J’s” Start Finish Start Finish Big Bang Continuous Improvement

87.
DevOps Kaizen: Let’sRecap! Service Delivery Metrics Kaizen Program Oversight Planning & Retrospectives Informs Informs Countermeasures & Blockers Ah-ha! OpsDev Remove Shorten Lead Time Feedback It’s about the business Make the work visible Continuous Improvement Actions to support improvement

88.
@damonedwards Damon Edwards damon@dtosolutions.com

DevOps Kaizen: Find and Fix What is Really Behind Your Problems

More Related Content

What's hot

Viewers also liked

Similar to DevOps Kaizen: Find and Fix What is Really Behind Your Problems

More from dev2ops

Recently uploaded

DevOps Kaizen: Find and Fix What is Really Behind Your Problems