SlideShare a Scribd company logo
1 of 25
$460Mn loss in 45 min, and similar stories. Will DevOps save our world?
Savinder Puri
@savinderpuri | savinderpuri@gmail.com
USD $460 million loss in 45 min! US $210 billion lost in one night!!
World War 3 just about averted!!!
When markets opened on 1st Aug at 9:30am, blood-bath happened in 45 min!
On 31st July 2012, Knight deployed a new version of software to Production
The new code re-purposed an obsoleted functionality “power-pegs”
SMARS is its high speed routing system for equity orders
Knight Capital WAS a global financial services firm.
It specialized with it’s high frequency trading algorithms
Total, USD $460 min lost.
All this in 45 min!
Knight Capital’s stock fell by over 70%.
It was officially termed as a “technology breakdown”
A manual rollback caused power-peg to being activated on all 8 SMARS!
This caused a major disruption in prices of 148 Companies listed on NYSE.
4 million child execution orders were inadvertently transacted
A technician forgot to copy the new code to one off 8 SMARS server
This activated the old PP, that moved stock prices up/down to verify trading behaviour
Issue 1: “Code deployed to only 7 off 8 SMARS servers”
Issue 2: “No review of deployments or gates”
Issue 2: “Notifications were not marked as errors, thus not taken seriously”
Issue 4: “Manual un-deploy carried out without taking cognizance of it's impact”
DevOps – a possible saviour!!!
• US $210 billion, or 86 perfect of cash in Asia's 3rd largest economy was
scrapped with this single decision.
We all saw this… on TV and in reality!
Scaling Reliability
Frequent Changes Out Experiment
This is what was expected, to react successfully to this “sudden change”
DevOps – a possible saviour!!!
Cloud adoption
Config-based deployments
Canary Deployments
Alfa-Beta testing
It’s a fast changing world, today!
Even with the world’s best tech, it still takes a human with the right values…
“We see ourselves as a technology company,
with a banking license”
“Why every Company Is A Technology Company”
“Every company is a tech company now
(yes, even Blue Apron)”
“Silicon Valley will be competing with us very soon.
We are ready”
All Tesla cars have the hardware to be driverless. All we need is software updates!
Biggest car company in the world. No. of cars owned?
No. of properties owned?
“Any customer can have a car painted any
colour that he wants so long as it is black.”
– Henry Ford
@savinderpuri | savinderpuri@gmail.com

More Related Content

Similar to $460 mn loss in 45 min, and similar stories. will devops save our world

How to prevent cyber terrorism taragana
How to prevent cyber terrorism  taraganaHow to prevent cyber terrorism  taragana
How to prevent cyber terrorism taragana
Gilles Sgro
 
DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?
DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?
DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?
Gene Kim
 

Similar to $460 mn loss in 45 min, and similar stories. will devops save our world (20)

21st Century Licensing Strategies
21st Century Licensing Strategies21st Century Licensing Strategies
21st Century Licensing Strategies
 
Applications of AI in Supply Chain Management: Hype versus Reality
Applications of AI in Supply Chain Management: Hype versus RealityApplications of AI in Supply Chain Management: Hype versus Reality
Applications of AI in Supply Chain Management: Hype versus Reality
 
DevOps and Cloud Native
DevOps and Cloud NativeDevOps and Cloud Native
DevOps and Cloud Native
 
Motion savvy deck (public)
Motion savvy deck (public)Motion savvy deck (public)
Motion savvy deck (public)
 
SE-4063, Leveraging Fingerprint Biometric Authentication to Streamline Secure...
SE-4063, Leveraging Fingerprint Biometric Authentication to Streamline Secure...SE-4063, Leveraging Fingerprint Biometric Authentication to Streamline Secure...
SE-4063, Leveraging Fingerprint Biometric Authentication to Streamline Secure...
 
Connecting Above the Cloud
Connecting Above the CloudConnecting Above the Cloud
Connecting Above the Cloud
 
How to prevent cyber terrorism taragana
How to prevent cyber terrorism  taraganaHow to prevent cyber terrorism  taragana
How to prevent cyber terrorism taragana
 
Hardware Startup Financing
Hardware Startup FinancingHardware Startup Financing
Hardware Startup Financing
 
The cyber security hype cycle is upon us
The cyber security hype cycle is upon usThe cyber security hype cycle is upon us
The cyber security hype cycle is upon us
 
“Real Time Machine Learning Architecture and Sentiment Analysis Applied to Fi...
“Real Time Machine Learning Architecture and Sentiment Analysis Applied to Fi...“Real Time Machine Learning Architecture and Sentiment Analysis Applied to Fi...
“Real Time Machine Learning Architecture and Sentiment Analysis Applied to Fi...
 
DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?
DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?
DevOps: Who Will Create $2.6 Trillion In Business Value Per Year?
 
Ppt ATD 24th_aug_final
Ppt ATD 24th_aug_finalPpt ATD 24th_aug_final
Ppt ATD 24th_aug_final
 
Startupfest 2012 - Coefficients of friction
Startupfest 2012 - Coefficients of frictionStartupfest 2012 - Coefficients of friction
Startupfest 2012 - Coefficients of friction
 
From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018From Duke of DevOps to Queen of Chaos - Api days 2018
From Duke of DevOps to Queen of Chaos - Api days 2018
 
Chaos Engineering: Why the World Needs More Resilient Systems
Chaos Engineering: Why the World Needs More Resilient SystemsChaos Engineering: Why the World Needs More Resilient Systems
Chaos Engineering: Why the World Needs More Resilient Systems
 
Crap. Your Big Data Kitchen Is Broken.
Crap. Your Big Data Kitchen Is Broken.Crap. Your Big Data Kitchen Is Broken.
Crap. Your Big Data Kitchen Is Broken.
 
20170613 iasa architecture - Tim Willoughby presentation
20170613   iasa architecture  - Tim Willoughby presentation20170613   iasa architecture  - Tim Willoughby presentation
20170613 iasa architecture - Tim Willoughby presentation
 
The Sky’s the Limit – The Rise of Machine Learnin
The Sky’s the Limit – The Rise of Machine LearninThe Sky’s the Limit – The Rise of Machine Learnin
The Sky’s the Limit – The Rise of Machine Learnin
 
Velocity Conference: Building a Scalable, Global SaaS Offering: Lessons from ...
Velocity Conference: Building a Scalable, Global SaaS Offering: Lessons from ...Velocity Conference: Building a Scalable, Global SaaS Offering: Lessons from ...
Velocity Conference: Building a Scalable, Global SaaS Offering: Lessons from ...
 
Security Opportunities A Silicon Valley VC Perspective
Security Opportunities  A Silicon Valley VC PerspectiveSecurity Opportunities  A Silicon Valley VC Perspective
Security Opportunities A Silicon Valley VC Perspective
 

Recently uploaded

Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Victor Rentea
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 

Recently uploaded (20)

Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Quantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingQuantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation Computing
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
Design Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptxDesign Guidelines for Passkeys 2024.pptx
Design Guidelines for Passkeys 2024.pptx
 
Choreo: Empowering the Future of Enterprise Software Engineering
Choreo: Empowering the Future of Enterprise Software EngineeringChoreo: Empowering the Future of Enterprise Software Engineering
Choreo: Empowering the Future of Enterprise Software Engineering
 
WSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering DevelopersWSO2's API Vision: Unifying Control, Empowering Developers
WSO2's API Vision: Unifying Control, Empowering Developers
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
JavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate GuideJavaScript Usage Statistics 2024 - The Ultimate Guide
JavaScript Usage Statistics 2024 - The Ultimate Guide
 
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
 
Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
الأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهالأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهله
 
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
 

$460 mn loss in 45 min, and similar stories. will devops save our world

  • 1. $460Mn loss in 45 min, and similar stories. Will DevOps save our world?
  • 2. Savinder Puri @savinderpuri | savinderpuri@gmail.com
  • 3. USD $460 million loss in 45 min! US $210 billion lost in one night!! World War 3 just about averted!!!
  • 4. When markets opened on 1st Aug at 9:30am, blood-bath happened in 45 min! On 31st July 2012, Knight deployed a new version of software to Production The new code re-purposed an obsoleted functionality “power-pegs” SMARS is its high speed routing system for equity orders Knight Capital WAS a global financial services firm. It specialized with it’s high frequency trading algorithms
  • 5. Total, USD $460 min lost. All this in 45 min! Knight Capital’s stock fell by over 70%. It was officially termed as a “technology breakdown” A manual rollback caused power-peg to being activated on all 8 SMARS! This caused a major disruption in prices of 148 Companies listed on NYSE. 4 million child execution orders were inadvertently transacted A technician forgot to copy the new code to one off 8 SMARS server This activated the old PP, that moved stock prices up/down to verify trading behaviour
  • 6.
  • 7.
  • 8.
  • 9. Issue 1: “Code deployed to only 7 off 8 SMARS servers”
  • 10. Issue 2: “No review of deployments or gates”
  • 11. Issue 2: “Notifications were not marked as errors, thus not taken seriously”
  • 12. Issue 4: “Manual un-deploy carried out without taking cognizance of it's impact”
  • 13. DevOps – a possible saviour!!!
  • 14. • US $210 billion, or 86 perfect of cash in Asia's 3rd largest economy was scrapped with this single decision.
  • 15. We all saw this… on TV and in reality!
  • 16. Scaling Reliability Frequent Changes Out Experiment This is what was expected, to react successfully to this “sudden change”
  • 17. DevOps – a possible saviour!!! Cloud adoption Config-based deployments Canary Deployments Alfa-Beta testing
  • 18. It’s a fast changing world, today!
  • 19. Even with the world’s best tech, it still takes a human with the right values…
  • 20. “We see ourselves as a technology company, with a banking license” “Why every Company Is A Technology Company” “Every company is a tech company now (yes, even Blue Apron)” “Silicon Valley will be competing with us very soon. We are ready”
  • 21. All Tesla cars have the hardware to be driverless. All we need is software updates!
  • 22. Biggest car company in the world. No. of cars owned?
  • 24. “Any customer can have a car painted any colour that he wants so long as it is black.” – Henry Ford