DevOpsGroup, profile picture
Uploaded byDevOpsGroup
534 views

Building for DevOps Success

The document discusses the significant failure of Knight Capital Group on August 1, 2012, when a code deployment error caused automated systems to send excessive stock orders, resulting in a $460 million loss within 45 minutes. It emphasizes the importance of proper deployment procedures and documentation to prevent such incidents, highlighting Knight's lack of a kill switch and inadequate review processes. Additionally, it touches on the principles of DevOps, suggesting that effective software delivery involves people, processes, and tools working together for better outcomes.

Embed presentation

Downloaded 15 times
1 / 114
2 / 114
3 / 114
4 / 114
5 / 114
6 / 114
7 / 114
8 / 114
9 / 114
10 / 114
11 / 114
12 / 114
13 / 114
14 / 114
15 / 114
16 / 114
17 / 114
18 / 114
19 / 114
20 / 114
21 / 114
22 / 114
23 / 114
24 / 114
25 / 114
26 / 114
27 / 114
28 / 114
29 / 114
30 / 114
31 / 114
32 / 114
33 / 114
34 / 114
35 / 114
36 / 114
37 / 114
38 / 114
39 / 114
40 / 114
41 / 114
42 / 114
43 / 114
44 / 114
45 / 114
46 / 114
47 / 114
48 / 114
49 / 114
50 / 114
51 / 114
52 / 114
53 / 114
54 / 114
55 / 114
56 / 114
57 / 114
58 / 114
59 / 114
60 / 114
61 / 114
62 / 114
63 / 114
64 / 114
65 / 114
66 / 114
67 / 114
68 / 114
69 / 114
70 / 114
71 / 114
72 / 114
73 / 114
74 / 114
75 / 114
76 / 114
77 / 114
78 / 114
79 / 114
80 / 114
81 / 114
82 / 114
83 / 114
84 / 114
85 / 114
86 / 114
87 / 114
88 / 114
89 / 114
90 / 114
91 / 114
92 / 114
93 / 114
94 / 114
95 / 114
96 / 114
97 / 114
98 / 114
99 / 114
100 / 114
101 / 114
102 / 114
103 / 114
104 / 114
105 / 114
106 / 114
107 / 114
108 / 114
109 / 114
110 / 114
111 / 114
112 / 114
113 / 114
114 / 114
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com 2016 BUILDING FOR DEVOPS SUCCESS
2 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr ABOUT ME • 20 years of web-scale engineering experience • Built e-commerce systems for Dell, Pizza Hut, KFC and more… • Dell generated $10 million/day. • The rest….$150 million in revenue every year. • Co-founder of the DevOpsGuys • http://www.devopsguys.com • @thedevmgr
3 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr DEVOPSGUYS ACCELERATE & TRANSFORM THE WAY ORGANISATIONS DELIVER SOFTWARE
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com SUCCESS! (BUT FIRST COMES FAILURE)
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr KNIGHT CAPITAL GROUP WALL
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr American global financial services company market making electronic execution institutional sales trading In 2012…the largest trader in US equities
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr 3.3 billion trades $21 billion A day!
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr On July 31, 2012 Knight had approximately $365 million in cash.
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Life is good!
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr NYSE announces plans to launch a new Retail Liquidity Program Enable retail customers to get price improvement for their stock orders
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Receives large orders from the trading platform and breaks them up into multiple smaller orders to find a buyer/seller match for the volume of shares. The larger the parent order, the more child orders would be generated. Photo Credit: Torben Bjørn Hansen via Compfight cc
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr 27 July 2016 “Power Peg” Knight stopped using this functionality in 2003 but the code was neither deleted nor deactivated. Photo Credit: Charles Crosbie via Compfight cc
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Code update repurposed an old flag that was used to activate the “Power Peg” functionality. The code was thoroughly tested and proven to work correctly and reliably.
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr What could possibly go wrong?
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Knight manually deploy updates to a limited number of servers per day 8 servers are “updated” Photo Credit: jovike via Compfight cc
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr If there is an SEC filing about your deployment something may have gone terribly wrong!
18 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr THE SEC FILING During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review. -- SEC Filing | Release No. 70694 | October 16, 2013 http://www.sec.gov/litigation/admin/2013/34-70694.pdf
19 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Missed one!
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr August 1st, the markets open Knight begin processing orders 7 servers begin processing orders correctly one does not! Photo Credit: harry harris via Compfight cc
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr “Power Peg” is back from the dead! Photo Credit: San Diego Shooter via Compfight cc
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Count the shares bought/sold against a parent order as child orders were executed Send the child orders for execution Stop once the parent share count has been fulfilled Photo Credit: marfis75 via Compfight cc
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr 27 July 2016 Cumulative tracking functionality moved to an earlier stage in the code execution And removed the count tracking from “Power Peg”Photo Credit: Vox Efx via Compfight cc
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr LOOP
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgrPhoto Credit: JD Hancock via Compfight cc
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Imagine what would happen if you had a system capable of sending automated, high-speed orders into the market without any tracking to see if enough orders had been executed
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr 27 July 2016 Photo Credit: Wooly Matt via Compfight cc OH s**T!
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgrPhoto Credit: fPat via Compfight cc OH s**T!
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr 9:30 AM – Markets Open 9:31 AM – Traders realise something is wrong 9:32 AM – Why hasn’t it stopped? Photo Credit: somewheregladlybeyond via Compfight cc
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr No kill switch Trading 8 million shares / minute +50% of the normal trading volume stocks +10% Photo Credit: Marshall Astor - Food Fetishist via Compfight cc
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr No documented procedures Start uninstalling the new code Removed the working code Amplified the issue Photo Credit: Jason Michael via Compfight cc Left the broken code
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr 45 minutes of trading 212 orders millions of child orders
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr 4 million transactions against 154 stocks 397 million shares
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr $460 million loss in 45-minutes Photo Credit: zachstern via Compfight cc
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Knight only has $365 million in cash
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr In 45-minutes Knight went from being the largest trader in US equities and a major market maker in the NYSE and NASDAQ Photo Credit: MattysFlicks via Compfight cc
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr NADA…. Bankrupt
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com WHAT’S THIS DEVOPS THING?
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr PEOPLE, PROCESS AND THE RIGHT TOOLS WORKING TOGETHER TO MAKEYOUR SOFTWARE DELIVERY LIFECYCLE FASTER AND MORE PREDICTABLE.
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr OPERABILITYScalability Deployability Resilience Monitoring Alerting Disaster Recovery Supportability Maintainability
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr CONTINUOUS DELIVERY + OPERABILITY = DEVOPS
44 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr KEY DEVOPS PATTERNS & PRACTICES Developer self-service Continuous monitoring Desired state management Monitor everything Integrated toolchains Infrastructure as code Collective Ownership Continuous Improvement Collaborative culture Continuous experimentation Full stack engineers Autonomous teams Integrated teams Joint meetings Job rotation Servant leadership Continuous integration Continuous testing Continuous delivery Minimum viable product Small batches Automated Builds Automated testing Release automation Canary rollouts Instrument everything Feature flags Version everything Test-driven development Technical debt Test everything Minimum viable process Test-driven deployment Common metrics Optimise flow Culture Process people technology
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com 8 DEVOPS LESSONS
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com #1 IDENTIFYTHE GOAL
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr +
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Company Department Team Individual
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr • Sprint • 2 weeks 6 18 • Month • 2 sprints Aspirational (60%) • Quarter • 3 months • Annual • 12 months 1 2 3 Vision
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr • Sprint • 2 weeks 6 18 • Month • 2 sprints Aspirational (60%) • Quarter • 3 months • Annual • 12 months 1 2 3 Positive (70%) VisionObjectives
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr • Sprint • 2 weeks 6 18 • Month • 2 sprints Aspirational (60%) • Quarter • 3 months • Annual • 12 months 1 2 3 Positive (70%) Considered (80%) VisionObjectivesPlan
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr • Sprint • 2 weeks 6 18 • Month • 2 sprints Aspirational (60%) • Quarter • 3 months • Annual • 12 months 1 2 3 Positive (70%) Considered (80%) Confident (95%) VisionPlanTasks Objectives
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr “SUCCESS IS NOT CHECKING A BOX. SUCCESS IS HAVING AN IMPACT. IFYOU COMPLETE ALL TASKS AND NOTHING EVER GETS BETTER,THAT'S NOT SUCCESS.” Christina Wodtke, OKR Coach
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com #2 BE AGILE
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Scrum Lean Kanban XP DSDM SAFe
57 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr SCRUM IS POPULAR Image source: State of Agile Survey, 2015
58 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Daily Stand-Ups Retrospectives Burndowns Velocity Iteration Planning Story Mapping Planning Poker
59 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Value people and how they communicate Measure progress through working software Collaborate with your customers Respond to change
60 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr THE MANIFESTO • Individuals and interactions over processes and tools • Working software over comprehensive documentation • Customer collaboration over contract negotiation • Responding to change over following a plan
61 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr BEING AGILE Image source: State of Agile Survey, 2015
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr LEARN TO INSPECT & ADAPT
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com #3 ALIGN TOVALUE
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Organization Roles Teams Cadence Taxonomy Plan Practices Alignment Autonomy “Let’s try to give our teams three things…. Autonomy, Mastery, Purpose”
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Alignment Every team and business tracks scenarios and features consistently. Autonomy Every team chooses how to manage stories and/or tasks
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr OPTIMISEYOUR ORGANISATION FOR PRODUCT DELIVERY
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com #4 DELIVER CONTINUOUSLY
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr TIME
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Continuous Delivery Frequent Release Events Waterfall Rare Release Events Smoother Effort Less Risk Effort Peaks High Risk Time Time Change Change
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr AUTOMATION IS IMPORTANT Comprehensive, fast and reliable test and deployment automation Trunk-based development & continuous integration Application code and app and system configuration all in version control Together, the factors on the left model continuous delivery which leads to… Lower levels of deployment pain Higher levels of IT performance (higher throughput & stability) Lower change fail rates Higher levels of org performance (productivity, market share, profitability)
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr https://xebialabs.com/periodic-table-of-devops-tools/
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr 15,000 engineers, working on 4,000+ projects All code is checked into one source tree (billions of files!) 5,500 code commits/day 75 million test cases/run daily
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr DEPLOYMENT PAIN CAN TELLYOU A LOT ABOUTYOUR IT PERFORMANCE
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com #5 MEASURE WHAT’S IMPORTANT
77 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr WHERE DO I START? • Start at the highest level • Is measurement happening? • Are metrics a thing? • Ask around!
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr START WITHYOUR IMPORTANTTHINGS TOYOUR TEAM
79 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr IDENTIFY A GOAL • Is there existing data? • Who are the key people? • What things are important to support this goal?
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr METRICS PROVIDE OPPORTUNITY FOR COMMUNICATION AND ALIGNMENT
81 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr IDENTIFY A GOAL: SHIP EVERY DAY • Is there existing data? • Who are the key people? • What things are important to support this goal?
82 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr IDENTIFY A GOAL: SHIP EVERY DAY • What things are important to support this goal? • Test Maturity • Unit tests • Component tests • Integration tests • Acceptance tests • Performance tests • Security tests
83 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr IDENTIFY A GOAL: SHIP EVERY DAY 0 1 2 3 4 5 6 Unit Integration Acceptance Performance Test Maturity Product A Product B Product C
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr WORKING SOFTWARE IS THE PRIMARY MEASURE OF PROGRESS.
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com #6 ACT ON FEEDBACK
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr THE DELIVERY TEAM MUST RECEIVE FEEDBACK AND ACT ON IT
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr INVOLVE EVERYBODY IN THE FEEDBACK PROCESS
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr BROADCASTTHE INFORMATION
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr FEED THE WASTE SNAKE
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr FEEDBACK MUST BE ACTED UPON
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr HOW MANYTIMES PER DAY ISTHE ANDON CORD PULLED IN A TYPICAL DAY AT A TOYOTA MANUFACTURING PLANT?
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr 3,500TIMES PER DAY
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr “SHIT HITS THE FAN, BUT THAT DOES NOT MEANYOU SHOULDN’T USE THE SAME DISCIPLINE.”
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com #7 BUILD TRUST
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr HIGHEST PREDICTOR OF PERFORMANCE The Study of Information Flow: A Personal Journey; Westrum, Ron
101 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr HOW WELL DOYOU IDENTIFYWITH? • On my team, information is actively sought. • On my team, failures are learning opportunities, and messengers of them are not punished. • On my team, responsibilities are shared. • On my team, cross-functional collaboration is encouraged and rewarded. • On my team, failure causes enquiry. • On my team, new ideas are welcomed.
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com #8 OPTIMISE FLOW
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr EVERYTHINGYOU DO SHOULD OPTIMISE FLOW
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr ANY OPTIMISATION NOT MADE AT THE CONSTRAINT IS AN ILLUSION
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr THROUGHPUT IS PROFITABLE, EFFICIENCY IS NOT
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr WHAT ISYOUR LEAD TIME FOR CHANGES?
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr HOW LONG DOES IT TAKE TO GO FROM CODE COMMITTEDTO CODE SUCCESSFULLY RUNNING IN PRODUCTION?
108 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr DEPLOYMENT LEADTIME PREDICTS • Ability for Dev and Ops to share a “common source of truth” • Effectiveness of our automated testing in the deployment pipeline • Ability to quickly deploy into production without causing chaos and disruption • Ability to detect and correct problems through monitoring • Ability for Dev and Ops to work together in a way that is “win / win” • How quickly developers can get feedback on their work • Testing, deploying, production outcomes, customer outcomes
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr Thousands of teams + Micro services Architecture + Continuous Delivery = 50 MILLION DEPLOYMENTS/YEAR 136K DEPLOYS/DAY
110 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr FLOW OPTIMISATION • Split into 2 teams • “Boss” has a stopwatch (on a phone) • Record first chip to “boss”, record last chip to “boss” • Developers job is to “flip the chip” (black dot up or down) • Stack chips in piles of 10 • Complete the batch before passing to next person
SPLIT INTO 2 TEAMS Team A Run # Time First Time Last 1 45 45 2 30 38 3 4 21 4 4 29 5 8 25 Team B Run # Time First Time Last 1 55 55 2 58 58 3 55 55 4 51 51 5 1.01 1.01
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr DEPLOY SMALLER CHANGES MORE FREQUENTLY
113 www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com @TheDevMgr 8 LESSONS 1. #IdentifyTheGoal 2. #BeAgile 3. #AlignToValue 4. #DeliverContinuously 5. #MeasureWhatsImportant 6. #ActOnFeedback 7. #BuildTrust 8. #OptimiseForFlow
www.devopsguys.com Phone: 0800 368 7378 | e-mail: team@devopsguys.com THANKYOU

More Related Content

PPTX
DevOps is the Answer... What was the question again? DevOps and Digital Trans...
byDevOpsGroup
 
PPTX
Maximising the value of the Cloud through DevOps
byDevOpsGroup
 
PPTX
DevOpsGuys FutureDecoded 2016 - is DevOps the Answer
byDevOpsGroup
 
PPTX
Five Ways Automation Has Increased Application Deployment and Changed Culture
byXebiaLabs
 
PPT
All Change how the economics of Cloud will make you think differently about Java
bySteve Poole
 
PPTX
DevOpsGuys Performance Testing with APM Tools workshop
byDevOpsGroup
 
PPTX
The benefits of using an APM solution while performance testing
byDevOpsGroup
 
PPTX
Top Lessons Learned From The DevOps Handbook
byXebiaLabs
 
DevOps is the Answer... What was the question again? DevOps and Digital Trans...
byDevOpsGroup
 
Maximising the value of the Cloud through DevOps
byDevOpsGroup
 
DevOpsGuys FutureDecoded 2016 - is DevOps the Answer
byDevOpsGroup
 
Five Ways Automation Has Increased Application Deployment and Changed Culture
byXebiaLabs
 
All Change how the economics of Cloud will make you think differently about Java
bySteve Poole
 
DevOpsGuys Performance Testing with APM Tools workshop
byDevOpsGroup
 
The benefits of using an APM solution while performance testing
byDevOpsGroup
 
Top Lessons Learned From The DevOps Handbook
byXebiaLabs
 

What's hot

PDF
Test Design for Fully Automated Build Architecture
byTechWell
 
PPTX
Continuous Testing: Preparing for DevOps
bySTePINForum
 
PDF
Strong Arm Your Tools
byTasktop
 
PPTX
The Devops Handbook
byHarish Kamugakudi Marimuthu
 
PDF
The Four Keys - Measuring DevOps Success
byDina Graves Portman
 
PPTX
WinOps Conf 2015 - John Rakowski - Militarise It for #DevOps success
byWinOps Conf
 
PDF
A Product Manager's Place in a DevOps World
byAtlassian
 
PDF
SOASTA mPulse update webinar
byCloudBees
 
PPTX
Taking Your Product Development to the Next Level with Full Stack
byOptimizely
 
PDF
Scaling IT Operations in a DevOps World
byAtlassian
 
PDF
Drupal and Devops , the Survey Results
byKris Buytaert
 
PDF
Shift left-devoxx-pl
byJohn Ferguson Smart Limited
 
PPTX
The What and Why of DevOps - DevOps Cardiff - May 2014
byDevOpsGroup
 
PPTX
Supercharging Optimizely Performance by Moving Decisions to the Edge
byOptimizely
 
PPTX
01 why of dev ops - devopsguys - magentys - final
byDevOpsGroup
 
PDF
Cloud-Native Builds & Deployments in Bitbucket Pipelines
byAtlassian
 
PDF
Scaling to 150,000 Builds a Month... and Beyond
byAtlassian
 
PDF
A DevOps Mario Developer Game Challenge with GRC
byBMK Lakshminarayanan
 
PDF
7 steps to pragmatic mobile testing
bySOASTA
 
PDF
Dev ops webinar 5 aug 15
byEdureka!
 
Test Design for Fully Automated Build Architecture
byTechWell
 
Continuous Testing: Preparing for DevOps
bySTePINForum
 
Strong Arm Your Tools
byTasktop
 
The Devops Handbook
byHarish Kamugakudi Marimuthu
 
The Four Keys - Measuring DevOps Success
byDina Graves Portman
 
WinOps Conf 2015 - John Rakowski - Militarise It for #DevOps success
byWinOps Conf
 
A Product Manager's Place in a DevOps World
byAtlassian
 
SOASTA mPulse update webinar
byCloudBees
 
Taking Your Product Development to the Next Level with Full Stack
byOptimizely
 
Scaling IT Operations in a DevOps World
byAtlassian
 
Drupal and Devops , the Survey Results
byKris Buytaert
 
Shift left-devoxx-pl
byJohn Ferguson Smart Limited
 
The What and Why of DevOps - DevOps Cardiff - May 2014
byDevOpsGroup
 
Supercharging Optimizely Performance by Moving Decisions to the Edge
byOptimizely
 
01 why of dev ops - devopsguys - magentys - final
byDevOpsGroup
 
Cloud-Native Builds & Deployments in Bitbucket Pipelines
byAtlassian
 
Scaling to 150,000 Builds a Month... and Beyond
byAtlassian
 
A DevOps Mario Developer Game Challenge with GRC
byBMK Lakshminarayanan
 
7 steps to pragmatic mobile testing
bySOASTA
 
Dev ops webinar 5 aug 15
byEdureka!
 

Viewers also liked

PDF
Continuous delivery for databases - Bristol DevOps Edition
byDevOpsGroup
 
PPTX
Moldeo por arena
byJorge de la Fuente Fernández
 
PPTX
Ponència logo & slogan
byGEMMMAD
 
PDF
S65 lecture 2
by맹 기완
 
PPTX
Angel i jassin
byGEMMMAD
 
PPT
Johanny hernandez. manierismo
byJohanny Hernandez
 
PPTX
Estilo Barroco
byJessy Yelamo
 
PPTX
SAPhybris: Brian Walker
byGamePlanConference
 
PPTX
Youth by Maximo D.Ramos
byChinly Ruth Alberto
 
PPTX
Desert ecosystem
byVinitha Chandra Sekar
 
PPTX
Fire alarm 2_d_radiant energy detectors
bygrantlerc
 
Continuous delivery for databases - Bristol DevOps Edition
byDevOpsGroup
 
Moldeo por arena
byJorge de la Fuente Fernández
 
Ponència logo & slogan
byGEMMMAD
 
S65 lecture 2
by맹 기완
 
Angel i jassin
byGEMMMAD
 
Johanny hernandez. manierismo
byJohanny Hernandez
 
Estilo Barroco
byJessy Yelamo
 
SAPhybris: Brian Walker
byGamePlanConference
 
Youth by Maximo D.Ramos
byChinly Ruth Alberto
 
Desert ecosystem
byVinitha Chandra Sekar
 
Fire alarm 2_d_radiant energy detectors
bygrantlerc
 

Similar to Building for DevOps Success

PPTX
WinOps Conf 2015 - Steve Thair - Why we need a DevOps on Windows Conference
byWinOps Conf
 
PPTX
GCP DevOps Certification Course in India | GCP DevOps
byranjithvisualpath44
 
PPTX
Devopsguys DevOps 101 for recruiters
byDevOpsGroup
 
PDF
Rackspace::Solve NYC - Solving for Rapid Customer Growth and Scale Through De...
byRackspace
 
PPTX
$460 mn loss in 45 min, and similar stories. will devops save our world
bySavinder Puri
 
PDF
Devops
byFernando Ike
 
PPTX
IDC Architecture 2016 - Devops and Open Source
by{code}
 
WinOps Conf 2015 - Steve Thair - Why we need a DevOps on Windows Conference
byWinOps Conf
 
GCP DevOps Certification Course in India | GCP DevOps
byranjithvisualpath44
 
Devopsguys DevOps 101 for recruiters
byDevOpsGroup
 
Rackspace::Solve NYC - Solving for Rapid Customer Growth and Scale Through De...
byRackspace
 
$460 mn loss in 45 min, and similar stories. will devops save our world
bySavinder Puri
 
Devops
byFernando Ike
 
IDC Architecture 2016 - Devops and Open Source
by{code}
 

More from DevOpsGroup

PPTX
DevOpsGroup Cloud Adoption Frameworks - agile south coast
byDevOpsGroup
 
PPTX
WinOps - Lessons learned from Enterprise DevOps with Microsoft technologies ...
byDevOpsGroup
 
PPTX
WinOps - Lessons learned from Enterprise DevOps with Microsoft technologies
byDevOpsGroup
 
PPTX
DevOpsGuys Scaling DevOps @ #CIOWaterCooler - June 2018
byDevOpsGroup
 
PPTX
DevOpsGuys - Cloud Adoption Frameworks - IPExpo April 2018
byDevOpsGroup
 
PPTX
The future of (Windows) operations #WinOps #DevOps
byDevOpsGroup
 
PPTX
Pester & PSScriptAnalyser - Power Test your PowerShell with PowerShell - Futu...
byDevOpsGroup
 
PPTX
Why Culture eats DevOps for Breakfast
byDevOpsGroup
 
PPTX
DevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
byDevOpsGroup
 
PPTX
Why #DevOps Transformation has to start with you
byDevOpsGroup
 
PPTX
DevOpsGuys - How to get started with DevOps - Redgate Webinar April 2017
byDevOpsGroup
 
PPTX
DevOpsGuys - Getting Started with DevOps - Github/Azure Webinar
byDevOpsGroup
 
PPTX
WinOps meetup April 2016 DevOps lessons from Microsoft \\Build\
byDevOpsGroup
 
PDF
Delivering DevOps Automation on Windows
byDevOpsGroup
 
PPTX
Continuous delivery for databases
byDevOpsGroup
 
PPTX
#DevOps in a windows world - @DevOpsGuys
byDevOpsGroup
 
PPTX
The WHY of DevOps (revised for DevOps Enthusiast Meetup London)
byDevOpsGroup
 
PDF
The 12 Days Of DevOps
byDevOpsGroup
 
PPTX
WorldwidePageTest demo @ #VelocityConf
byDevOpsGroup
 
PPTX
Managing Performance in the Cloud
byDevOpsGroup
 
DevOpsGroup Cloud Adoption Frameworks - agile south coast
byDevOpsGroup
 
WinOps - Lessons learned from Enterprise DevOps with Microsoft technologies ...
byDevOpsGroup
 
WinOps - Lessons learned from Enterprise DevOps with Microsoft technologies
byDevOpsGroup
 
DevOpsGuys Scaling DevOps @ #CIOWaterCooler - June 2018
byDevOpsGroup
 
DevOpsGuys - Cloud Adoption Frameworks - IPExpo April 2018
byDevOpsGroup
 
The future of (Windows) operations #WinOps #DevOps
byDevOpsGroup
 
Pester & PSScriptAnalyser - Power Test your PowerShell with PowerShell - Futu...
byDevOpsGroup
 
Why Culture eats DevOps for Breakfast
byDevOpsGroup
 
DevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
byDevOpsGroup
 
Why #DevOps Transformation has to start with you
byDevOpsGroup
 
DevOpsGuys - How to get started with DevOps - Redgate Webinar April 2017
byDevOpsGroup
 
DevOpsGuys - Getting Started with DevOps - Github/Azure Webinar
byDevOpsGroup
 
WinOps meetup April 2016 DevOps lessons from Microsoft \\Build\
byDevOpsGroup
 
Delivering DevOps Automation on Windows
byDevOpsGroup
 
Continuous delivery for databases
byDevOpsGroup
 
#DevOps in a windows world - @DevOpsGuys
byDevOpsGroup
 
The WHY of DevOps (revised for DevOps Enthusiast Meetup London)
byDevOpsGroup
 
The 12 Days Of DevOps
byDevOpsGroup
 
WorldwidePageTest demo @ #VelocityConf
byDevOpsGroup
 
Managing Performance in the Cloud
byDevOpsGroup
 

Recently uploaded

PDF
Transcript: Escape from the Forbidden Zone: Smuggling green and inclusive tec...
byBookNet Canada
 
PPTX
GenerationAI Paris 2025 | AI in Tech: Beyond Expectations, Into Execution
byapidays
 
PDF
HOW TO OVERCOME THE THREATS OF ARTIFICIAL INTELLIGENCE AGAINST HUMANITY.pdf
byFaga1939
 
PDF
How does MES(Manufacturing Execution System) work?
byakipii ogaoga
 
PDF
Constraint Collapse and Fidelity Decay in Scaled Language Models
bySemantic Fidelity Lab | A. Jacobs
 
PDF
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
PDF
GTM-and-Sales-Plan for a cyber security product
byAshish Jangir
 
PDF
How AI Can Help Platform Engineers Build Better Platforms
byAll Things Open
 
PPTX
Tech Trends 2026: AI Agents, Quantum Computing, Robotics & Cybersecurity
byridwansassman
 
PDF
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 
PDF
Explaining the flow of purpose-specific BOM
byakipii ogaoga
 
PPTX
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
PDF
Digital Twin in IBM for Accelerated Discovery of Climate & Sustainability, K...
byMichiaki Tatsubori
 
PPTX
Workflow and decision Automation with Flowable
bychakib ch
 
PPTX
Microsoft Azure News - February 2026 - BAUG
byDaniel Toomey
 
PDF
TrustArc Webinar - From Trends to Action: Fitting AI Governance into Privacy Ops
byTrustArc
 
PDF
Empower your IT team with cloud-based PC management using Dell Management Por...
byPrincipled Technologies
 
PDF
February 2026 Patch Tuesday hosted by Chris Goettl and Todd Schell
byIvanti
 
PDF
Chapter 5 Security Mechanisms iin computer security.pdf
byGetnet Tigabie Askale -(GM)
 
PDF
GenerationAI_Paris_2025_Architecting_Intelligence.pdf
byapidays
 
Transcript: Escape from the Forbidden Zone: Smuggling green and inclusive tec...
byBookNet Canada
 
GenerationAI Paris 2025 | AI in Tech: Beyond Expectations, Into Execution
byapidays
 
HOW TO OVERCOME THE THREATS OF ARTIFICIAL INTELLIGENCE AGAINST HUMANITY.pdf
byFaga1939
 
How does MES(Manufacturing Execution System) work?
byakipii ogaoga
 
Constraint Collapse and Fidelity Decay in Scaled Language Models
bySemantic Fidelity Lab | A. Jacobs
 
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
GTM-and-Sales-Plan for a cyber security product
byAshish Jangir
 
How AI Can Help Platform Engineers Build Better Platforms
byAll Things Open
 
Tech Trends 2026: AI Agents, Quantum Computing, Robotics & Cybersecurity
byridwansassman
 
UiPath Automation Developer Associate Training Series 2026 - Session 3
byDianaGray10
 
Explaining the flow of purpose-specific BOM
byakipii ogaoga
 
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
Digital Twin in IBM for Accelerated Discovery of Climate & Sustainability, K...
byMichiaki Tatsubori
 
Workflow and decision Automation with Flowable
bychakib ch
 
Microsoft Azure News - February 2026 - BAUG
byDaniel Toomey
 
TrustArc Webinar - From Trends to Action: Fitting AI Governance into Privacy Ops
byTrustArc
 
Empower your IT team with cloud-based PC management using Dell Management Por...
byPrincipled Technologies
 
February 2026 Patch Tuesday hosted by Chris Goettl and Todd Schell
byIvanti
 
Chapter 5 Security Mechanisms iin computer security.pdf
byGetnet Tigabie Askale -(GM)
 
GenerationAI_Paris_2025_Architecting_Intelligence.pdf
byapidays
 

Building for DevOps Success

  • 1.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com 2016 BUILDING FOR DEVOPS SUCCESS
  • 2.
    2 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr ABOUT ME • 20 years of web-scale engineering experience • Built e-commerce systems for Dell, Pizza Hut, KFC and more… • Dell generated $10 million/day. • The rest….$150 million in revenue every year. • Co-founder of the DevOpsGuys • http://www.devopsguys.com • @thedevmgr
  • 3.
    3 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr DEVOPSGUYS ACCELERATE & TRANSFORM THE WAY ORGANISATIONS DELIVER SOFTWARE
  • 4.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr
  • 5.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com SUCCESS! (BUT FIRST COMES FAILURE)
  • 6.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr KNIGHT CAPITAL GROUP WALL
  • 7.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr American global financial services company market making electronic execution institutional sales trading In 2012…the largest trader in US equities
  • 8.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr 3.3 billion trades $21 billion A day!
  • 9.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr On July 31, 2012 Knight had approximately $365 million in cash.
  • 10.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Life is good!
  • 11.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr NYSE announces plans to launch a new Retail Liquidity Program Enable retail customers to get price improvement for their stock orders
  • 12.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Receives large orders from the trading platform and breaks them up into multiple smaller orders to find a buyer/seller match for the volume of shares. The larger the parent order, the more child orders would be generated. Photo Credit: Torben Bjørn Hansen via Compfight cc
  • 13.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr 27 July 2016 “Power Peg” Knight stopped using this functionality in 2003 but the code was neither deleted nor deactivated. Photo Credit: Charles Crosbie via Compfight cc
  • 14.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Code update repurposed an old flag that was used to activate the “Power Peg” functionality. The code was thoroughly tested and proven to work correctly and reliably.
  • 15.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr What could possibly go wrong?
  • 16.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Knight manually deploy updates to a limited number of servers per day 8 servers are “updated” Photo Credit: jovike via Compfight cc
  • 17.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr If there is an SEC filing about your deployment something may have gone terribly wrong!
  • 18.
    18 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr THE SEC FILING During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review. -- SEC Filing | Release No. 70694 | October 16, 2013 http://www.sec.gov/litigation/admin/2013/34-70694.pdf
  • 19.
    19 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Missed one!
  • 20.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr August 1st, the markets open Knight begin processing orders 7 servers begin processing orders correctly one does not! Photo Credit: harry harris via Compfight cc
  • 21.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr “Power Peg” is back from the dead! Photo Credit: San Diego Shooter via Compfight cc
  • 22.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Count the shares bought/sold against a parent order as child orders were executed Send the child orders for execution Stop once the parent share count has been fulfilled Photo Credit: marfis75 via Compfight cc
  • 23.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr 27 July 2016 Cumulative tracking functionality moved to an earlier stage in the code execution And removed the count tracking from “Power Peg”Photo Credit: Vox Efx via Compfight cc
  • 24.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr LOOP
  • 25.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgrPhoto Credit: JD Hancock via Compfight cc
  • 26.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Imagine what would happen if you had a system capable of sending automated, high-speed orders into the market without any tracking to see if enough orders had been executed
  • 27.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr 27 July 2016 Photo Credit: Wooly Matt via Compfight cc OH s**T!
  • 28.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgrPhoto Credit: fPat via Compfight cc OH s**T!
  • 29.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr
  • 30.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr 9:30 AM – Markets Open 9:31 AM – Traders realise something is wrong 9:32 AM – Why hasn’t it stopped? Photo Credit: somewheregladlybeyond via Compfight cc
  • 31.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr No kill switch Trading 8 million shares / minute +50% of the normal trading volume stocks +10% Photo Credit: Marshall Astor - Food Fetishist via Compfight cc
  • 32.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr No documented procedures Start uninstalling the new code Removed the working code Amplified the issue Photo Credit: Jason Michael via Compfight cc Left the broken code
  • 33.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr 45 minutes of trading 212 orders millions of child orders
  • 34.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr 4 million transactions against 154 stocks 397 million shares
  • 35.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr $460 million loss in 45-minutes Photo Credit: zachstern via Compfight cc
  • 36.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Knight only has $365 million in cash
  • 37.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr In 45-minutes Knight went from being the largest trader in US equities and a major market maker in the NYSE and NASDAQ Photo Credit: MattysFlicks via Compfight cc
  • 38.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr NADA…. Bankrupt
  • 39.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com WHAT’S THIS DEVOPS THING?
  • 40.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr PEOPLE, PROCESS AND THE RIGHT TOOLS WORKING TOGETHER TO MAKEYOUR SOFTWARE DELIVERY LIFECYCLE FASTER AND MORE PREDICTABLE.
  • 41.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr
  • 42.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr OPERABILITYScalability Deployability Resilience Monitoring Alerting Disaster Recovery Supportability Maintainability
  • 43.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr CONTINUOUS DELIVERY + OPERABILITY = DEVOPS
  • 44.
    44 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr KEY DEVOPS PATTERNS & PRACTICES Developer self-service Continuous monitoring Desired state management Monitor everything Integrated toolchains Infrastructure as code Collective Ownership Continuous Improvement Collaborative culture Continuous experimentation Full stack engineers Autonomous teams Integrated teams Joint meetings Job rotation Servant leadership Continuous integration Continuous testing Continuous delivery Minimum viable product Small batches Automated Builds Automated testing Release automation Canary rollouts Instrument everything Feature flags Version everything Test-driven development Technical debt Test everything Minimum viable process Test-driven deployment Common metrics Optimise flow Culture Process people technology
  • 45.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com 8 DEVOPS LESSONS
  • 46.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com #1 IDENTIFYTHE GOAL
  • 47.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr
  • 48.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr +
  • 49.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Company Department Team Individual
  • 50.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr • Sprint • 2 weeks 6 18 • Month • 2 sprints Aspirational (60%) • Quarter • 3 months • Annual • 12 months 1 2 3 Vision
  • 51.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr • Sprint • 2 weeks 6 18 • Month • 2 sprints Aspirational (60%) • Quarter • 3 months • Annual • 12 months 1 2 3 Positive (70%) VisionObjectives
  • 52.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr • Sprint • 2 weeks 6 18 • Month • 2 sprints Aspirational (60%) • Quarter • 3 months • Annual • 12 months 1 2 3 Positive (70%) Considered (80%) VisionObjectivesPlan
  • 53.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr • Sprint • 2 weeks 6 18 • Month • 2 sprints Aspirational (60%) • Quarter • 3 months • Annual • 12 months 1 2 3 Positive (70%) Considered (80%) Confident (95%) VisionPlanTasks Objectives
  • 54.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr “SUCCESS IS NOT CHECKING A BOX. SUCCESS IS HAVING AN IMPACT. IFYOU COMPLETE ALL TASKS AND NOTHING EVER GETS BETTER,THAT'S NOT SUCCESS.” Christina Wodtke, OKR Coach
  • 55.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com #2 BE AGILE
  • 56.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Scrum Lean Kanban XP DSDM SAFe
  • 57.
    57 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr SCRUM IS POPULAR Image source: State of Agile Survey, 2015
  • 58.
    58 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Daily Stand-Ups Retrospectives Burndowns Velocity Iteration Planning Story Mapping Planning Poker
  • 59.
    59 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Value people and how they communicate Measure progress through working software Collaborate with your customers Respond to change
  • 60.
    60 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr THE MANIFESTO • Individuals and interactions over processes and tools • Working software over comprehensive documentation • Customer collaboration over contract negotiation • Responding to change over following a plan
  • 61.
    61 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr BEING AGILE Image source: State of Agile Survey, 2015
  • 62.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr LEARN TO INSPECT & ADAPT
  • 63.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com #3 ALIGN TOVALUE
  • 64.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr
  • 65.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Organization Roles Teams Cadence Taxonomy Plan Practices Alignment Autonomy “Let’s try to give our teams three things…. Autonomy, Mastery, Purpose”
  • 66.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Alignment Every team and business tracks scenarios and features consistently. Autonomy Every team chooses how to manage stories and/or tasks
  • 67.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr OPTIMISEYOUR ORGANISATION FOR PRODUCT DELIVERY
  • 68.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com #4 DELIVER CONTINUOUSLY
  • 69.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr TIME
  • 70.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Continuous Delivery Frequent Release Events Waterfall Rare Release Events Smoother Effort Less Risk Effort Peaks High Risk Time Time Change Change
  • 71.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr
  • 72.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr AUTOMATION IS IMPORTANT Comprehensive, fast and reliable test and deployment automation Trunk-based development & continuous integration Application code and app and system configuration all in version control Together, the factors on the left model continuous delivery which leads to… Lower levels of deployment pain Higher levels of IT performance (higher throughput & stability) Lower change fail rates Higher levels of org performance (productivity, market share, profitability)
  • 73.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr https://xebialabs.com/periodic-table-of-devops-tools/
  • 74.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr 15,000 engineers, working on 4,000+ projects All code is checked into one source tree (billions of files!) 5,500 code commits/day 75 million test cases/run daily
  • 75.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr DEPLOYMENT PAIN CAN TELLYOU A LOT ABOUTYOUR IT PERFORMANCE
  • 76.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com #5 MEASURE WHAT’S IMPORTANT
  • 77.
    77 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr WHERE DO I START? • Start at the highest level • Is measurement happening? • Are metrics a thing? • Ask around!
  • 78.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr START WITHYOUR IMPORTANTTHINGS TOYOUR TEAM
  • 79.
    79 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr IDENTIFY A GOAL • Is there existing data? • Who are the key people? • What things are important to support this goal?
  • 80.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr METRICS PROVIDE OPPORTUNITY FOR COMMUNICATION AND ALIGNMENT
  • 81.
    81 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr IDENTIFY A GOAL: SHIP EVERY DAY • Is there existing data? • Who are the key people? • What things are important to support this goal?
  • 82.
    82 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr IDENTIFY A GOAL: SHIP EVERY DAY • What things are important to support this goal? • Test Maturity • Unit tests • Component tests • Integration tests • Acceptance tests • Performance tests • Security tests
  • 83.
    83 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr IDENTIFY A GOAL: SHIP EVERY DAY 0 1 2 3 4 5 6 Unit Integration Acceptance Performance Test Maturity Product A Product B Product C
  • 84.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr WORKING SOFTWARE IS THE PRIMARY MEASURE OF PROGRESS.
  • 85.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com #6 ACT ON FEEDBACK
  • 86.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr THE DELIVERY TEAM MUST RECEIVE FEEDBACK AND ACT ON IT
  • 87.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr
  • 88.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr INVOLVE EVERYBODY IN THE FEEDBACK PROCESS
  • 89.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr
  • 90.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr BROADCASTTHE INFORMATION
  • 91.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr
  • 92.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr FEED THE WASTE SNAKE
  • 93.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr
  • 94.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr FEEDBACK MUST BE ACTED UPON
  • 95.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr
  • 96.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr HOW MANYTIMES PER DAY ISTHE ANDON CORD PULLED IN A TYPICAL DAY AT A TOYOTA MANUFACTURING PLANT?
  • 97.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr 3,500TIMES PER DAY
  • 98.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr “SHIT HITS THE FAN, BUT THAT DOES NOT MEANYOU SHOULDN’T USE THE SAME DISCIPLINE.”
  • 99.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com #7 BUILD TRUST
  • 100.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr HIGHEST PREDICTOR OF PERFORMANCE The Study of Information Flow: A Personal Journey; Westrum, Ron
  • 101.
    101 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr HOW WELL DOYOU IDENTIFYWITH? • On my team, information is actively sought. • On my team, failures are learning opportunities, and messengers of them are not punished. • On my team, responsibilities are shared. • On my team, cross-functional collaboration is encouraged and rewarded. • On my team, failure causes enquiry. • On my team, new ideas are welcomed.
  • 102.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com #8 OPTIMISE FLOW
  • 103.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr EVERYTHINGYOU DO SHOULD OPTIMISE FLOW
  • 104.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr ANY OPTIMISATION NOT MADE AT THE CONSTRAINT IS AN ILLUSION
  • 105.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr THROUGHPUT IS PROFITABLE, EFFICIENCY IS NOT
  • 106.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr WHAT ISYOUR LEAD TIME FOR CHANGES?
  • 107.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr HOW LONG DOES IT TAKE TO GO FROM CODE COMMITTEDTO CODE SUCCESSFULLY RUNNING IN PRODUCTION?
  • 108.
    108 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr DEPLOYMENT LEADTIME PREDICTS • Ability for Dev and Ops to share a “common source of truth” • Effectiveness of our automated testing in the deployment pipeline • Ability to quickly deploy into production without causing chaos and disruption • Ability to detect and correct problems through monitoring • Ability for Dev and Ops to work together in a way that is “win / win” • How quickly developers can get feedback on their work • Testing, deploying, production outcomes, customer outcomes
  • 109.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr Thousands of teams + Micro services Architecture + Continuous Delivery = 50 MILLION DEPLOYMENTS/YEAR 136K DEPLOYS/DAY
  • 110.
    110 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr FLOW OPTIMISATION • Split into 2 teams • “Boss” has a stopwatch (on a phone) • Record first chip to “boss”, record last chip to “boss” • Developers job is to “flip the chip” (black dot up or down) • Stack chips in piles of 10 • Complete the batch before passing to next person
  • 111.
    SPLIT INTO 2TEAMS Team A Run # Time First Time Last 1 45 45 2 30 38 3 4 21 4 4 29 5 8 25 Team B Run # Time First Time Last 1 55 55 2 58 58 3 55 55 4 51 51 5 1.01 1.01
  • 112.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr DEPLOY SMALLER CHANGES MORE FREQUENTLY
  • 113.
    113 www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com @TheDevMgr 8 LESSONS 1. #IdentifyTheGoal 2. #BeAgile 3. #AlignToValue 4. #DeliverContinuously 5. #MeasureWhatsImportant 6. #ActOnFeedback 7. #BuildTrust 8. #OptimiseForFlow
  • 114.
    www.devopsguys.com Phone: 0800 3687378 | e-mail: team@devopsguys.com THANKYOU