Continuous Delivery for Financial Services from 51zero.
•Download as PPTX, PDF•
0 likes•343 views
Financial Services carry too much risk it’s time to embrace Continuous Delivery
Recommended
Recommended
More Related Content
Similar to Continuous Delivery for Financial Services from 51zero.
Similar to Continuous Delivery for Financial Services from 51zero. (20)
Recently uploaded
Recently uploaded (20)
Continuous Delivery for Financial Services from 51zero.
- 1. Gary Frost – 51zero Ltd1 Financial Services carry too much risk it’s time to embrace Continuous Delivery Gary Frost 51zero Ltd www.51zero.com linkedin.com/company/51zero facebook.com/51zero/ @51zeroLtd
- 2. Gary Frost – 51zero Ltd2 About me Gary Frost 51zero Ltd • Managing Director 51zero Ltd • ~20 years investment banking IT • 10 banks • 4 countries • Battle scarred Agile Pragmatist linkedin.com/company/51zero facebook.com/51zero/ @51zeroLtd
- 3. Gary Frost – 51zero Ltd3 Agility Operations Security Defects Financial Services Carry To Much Software Risk Accidental or Deliberate code defects Performance degradation System outages Failed deployments Information theft, hacking Time to market Delivery/change velocity Development cost
- 4. Gary Frost – 51zero Ltd4 Development in the late 90’s/Early 2000’s Highly Agile Direct engagement with business, high velocity High Velocity – Risk Risk Little testing, unlimited access, no change control Business Development Velocity Risk Low risk High risk High Velocity Low Velocity
- 5. Gary Frost – 51zero Ltd5 A history of some other failures in the 1990’s/2000’s • 1996 – First National Bank of Chicago - $746 billion • Software bug • > $900m into 823 customer accounts • All funds recovered and systems corrected • Largest known errors • 2004 – Millions of customers effected as major US bank suffers software error in transaction processing - $100m loss • 2005 – All trading halted in Major Asian exchange due to a software glitch
- 6. Gary Frost – 51zero Ltd6 Sarbanes-Oxley IT Operations Control Comprehensive Audit Trails IT Process Controls SOX Controls from an IT Perspective SOX
- 7. Gary Frost – 51zero Ltd7 Segregation of Duties Business Person IT Governance board Developers Architect / Security / Others System Admin / Ops Identify Requirement Authorize & Approve Design / Code Review / Inspect / Approve Implement in Production Segregation of duties , no person should perform more than one of these roles
- 8. Gary Frost – 51zero Ltd8 Change ControlOperations Information Classification SecurityDataArchitecture Change control reviews prior to deployment (After the change has been agreed, developed and tested) • 2 week process • 6+ for major changes • Time intensive/Laborious
- 9. Gary Frost – 51zero Ltd9 The Cost Of This Process • Increased Staff • Time in meetings/review • Architectural inflexibility • Lower release frequency • Reduced Agility • Talent Acquisition/Retention
- 10. Gary Frost – 51zero Ltd10 Low Velocity – Low Risk? Velocity Risk Low risk High risk High Velocity Low Velocity ?
- 11. Gary Frost – 51zero Ltd11 Low Velocity – Low Risk? Velocity Risk Low risk High risk High Velocity Low Velocity
- 12. Gary Frost – 51zero Ltd12 Failures since SOX (and other Regulations) • 2012 – RBS/Natwest/Ulster bank – Payments disrupted – > 12 Millions of accounts impacted – Caused by software update – Bank fined £54 million by regulators – 3 similar episodes since 2012 • 2012 – Knight Capital – New feature deployed to 7 of 8 servers – Orders Routed to 8th server caused ’test’ price volatility in live market – $440 million lost in 45 minutes • 2015 – Barclays payments mobile banking glitch & card payments • 2015 – HSBC payments failed before bank holiday
- 13. Gary Frost – 51zero Ltd13 Why are we still failing? If we’ve put in place all these safety measures, at such cost, then why do we still get so many failures?
- 14. Gary Frost – 51zero Ltd14 Unmanageable Change Review Process • Thousands of systems (n) • Who can understand E2E • Many releases per system per year (m) • = unmanageable change review requirement (n*m)
- 15. Gary Frost – 51zero Ltd15 How Do We Solve This?
- 16. Gary Frost – 51zero Ltd16 • All code in Modern VCS • Pull Requests/Code review • Code quality checks • Build on change • Binary Repository • Security Testing • Software / License Testing • Performance Testing • Integration testing • Consumer Driven Contracts Testing • Infrastructure as Code • One click deploy • Environment provisioning • Containerization • Orchestration / Operating System for Datacenter • Metadata & data flows • Generated Architecture documents • Living / Breathing enterprise view Adopting Continuous Delivery Source Code & Build Automate Testing Automate Deployment Living Architecture 01 02 03 04
- 17. Gary Frost – 51zero Ltd17 High Maturity Development Test All The (F*) Time & Automate All The (F*) Things Build , Provision, Deploy Across envs E2E / Integration Testing Security / Data Testing Performance / Destructive Testing Promotion
- 18. Gary Frost – 51zero Ltd18 Change Review Teams Under extreme time pressures to review all changes Incentivized to manage risks Default position tends to “Unsafe / Stop” Development Teams Under extreme pressure to implement features Incentivized to delivery quickly Default position tends to be ”Safe / Go” The Biggest Blocker Silos lead to conflicting incentives
- 19. Gary Frost – 51zero Ltd19 Highly Collaborate Environments No more Us and Them Information Sec: Let’s write a test for secret data being written into the cloud Security: Let’s write a test the simulates a DDOS Ops: Let’s write a test that simulates a rack going offline Dev: Let’s write the system so it can be tested, and so that WE can write the tests
- 20. Gary Frost – 51zero Ltd20 Look To High Performing Industry Leaders Spotify, Google, HP, Twitter, Etsy, AirBnB, …
- 21. Gary Frost – 51zero Ltd21 Move to High Velocity – Low Risk Velocity Risk Low risk High risk High Velocity Low Velocity High level of Continuous Delivery Maturity
- 22. Gary Frost – 51zero Ltd22 Not Trivial • Not “Cheap and Easy” (we would have already done it) • Not just about tool choice • Not about hiring “DevOps” engineers • Requires process change and organizational change • There are isolated success stories – find them, study them, repeat them @ scale
- 23. Gary Frost – 51zero Ltd23 FinTech Disruption + + + + Well Funded No Legacy Moving Fast Innovative ideas Financial Service Disruption FinTech companies are forming at a rapid pace, they are finding their place in the market, they are attracting customers and funding. FinTech has now become a major risk to established Financial Service companies.
- 24. Gary Frost – 51zero Ltd24 Evolve Or Face Extinction “It is not necessary to change. Survival is not mandatory. W. Edwards Deming” The Future of Financial Services IT
- 25. Gary Frost – 51zero Ltd25 Thank You!!! Gary Frost 51zero Ltd www.51zero.com linkedin.com/company/51zero facebook.com/51zero/ @51zeroLtd I probably ran out of time, I ’m passionate about CD/DevOps, Big Data and Financial Services, so catch me after or contact me on the channels below. If I didn’t, any questions?
Editor's Notes
- At 51zero we develop software, especially big data software, for Financial service organisations and we use Continuous Delivery in our development practices. Gary Frost is the Managing Director of 51zero and has spent nearly 20 years in software development for Financial services.
- This presentation focuses on software risk in Financial services, the risk of defects in the code, the risk of a security vulnerability, of malicious code change, of information theft, of a performance degradation or a wholesale systems outage; the risk of a software change or deployment which can introduce or cause any of these.
- When I started in investment banking IT nearly 20 years ago, I was working for a relatively small Investment Bank in Australia, we used to work directly with the business and often we would respond to their requests and make changes to the system within hours, we would make changes in development environments and push them rapidly into production, we would log in and push our development changes ourselves. There was very little in the way of change control. This did allow us to be incredibly responsive and allowed a lot of innovation However there was little in the way of controllers around developer access to production, we had direct access to production systems and effectively no change control. It was indeed the wild west.
- There are many examples of major software failures in Financial Services during the 1990’s/2000’s
- Sarbanes-Oxley (commonly called as SOX) came about because of a number of large scale embezzlement cases and accounting fraud cases. It focuses on regulations and accounting practices put in place to avoid accounting scandals. From an IT perspective it requires Comprehensive Audit trials IT Controls, especially in key systems such as financial processing, payments, payroll, general ledgers IT Operations controls, access controls
- But SOX also discussed segregation of duties, primarily within business processes, but a number IT Audits found that an unexpectedly high proportion of their Sarbanes-Oxley internal control issues came from IT. As such it has become commonly accepted to implement separation of duties in large IT organizations so that the same person or organizations performs only one of the following roles: Identification of a requirement (or change request); e.g. a business person Authorization and approval; e.g. an IT governance board or manager Design and development; e.g. a developer Review, inspection and approval; e.g. another developer or architect. Implementation in production; typically a software change or system administrator.
- The process of getting a change into production, once the change management team have worked with the business and the technology team to design the change and it’s been fully UAT’ed, involves a lot of steps. It’s not uncommon to see a flow like Architectural review from a central team - standard patterns, authorized software used, system simplification/rationalisation. Data and database review from data base administration team Software Security review to look network vulnerabilities, malicious code, clear text passwords, encryption vulnerabilities Information security review to ensure sensitive data is securely stored, appropriate level of control for the type of information Operations review - review the support documentation, have training on support, and ensure the release instructions are comprehensive Change coordination - signoff from all reviewers, upstream/downstream systems owners, schedule change Then the operations team can deploy the change, working from a run book, often 10+ pages, written in a word document.
- It should be obviously that these processes come at as significant cost. Staff - significant increase in staff for all these change/review bodies Time - lots of additional documentation, meetings and review Architectural/Technological inflexibility Lower release frequency - no one wants to go through this too regularly, so changes stack up Inevitably leading to a lack of agility Talent acquisition/retention - because, it’s a competitive market, and those that can work at innovative, lean companies using the best technologies and development methodologies, tend to want to do do
- So we have traded speed which caused ticking time bombs, to a slower (sloth like) pace which is safer. These measure must have massively reduced the risk of system issues, right? Wrong!
- Wrong! From my research and experience there has only been a marginal reduction in risk, but a very significant increase in velocity.
- Looking at some more recent failures. 2012 - “The worst banking meltdown to date hit millions of customers of RBS, NatWest and Ulster Bank, locking them out of their accounts for days, and in the case of Ulster Bank customers, for weeks. The glitch, the result of a software patch, lead the the CA7 batch process scheduler ended with 12 million customer accounts being frozen - The Bank was fined £50 million by regulators but the episode is understood to have cost the bank more than £100 million.” RBS/Natwest have had at least 3 similar episodes since 2012. Of of the most destructive recent software failues was Knight capital. 2012, Knight Capital deployed untested software to a production environment which contained an obsolete function. The incident happened due to a technician forgetting to copy the new Retail Liquidity Program (RLP) code to one of the eight SMARS computer servers […] when released into production, Knight's trading activities caused a major disruption in the prices of 148 companies listed at the New York Stock Exchange, thus, for example, shares of Wizzard Software Corporation went from $3.50 to $14.76. Knight Capital took a pre-tax loss of $440 million” 2015 Barclays has experienced a number of technical glitches, which have caused problems with mobile banking and card payments for some customers. The technical faults saw online and mobile banking services go down for periods throughout Friday afternoon, and some customers had issues withdrawing cash or making payments with their cards 2015: “Thousands of people have been left without their salaries because of an IT glitch at HSBC that means employers who use its business banking accounts cannot make payments. Some 275,000 individual payments failed to go through on Friday leaving potentially hundreds of thousands of people without their pay on the Friday before the bank holiday weekend.”
- Well, first let’s take a quick peek at a not atypical system architecture. This diagram that was shown at an event some time ago. It depicts all the individual processing components and their interdependencies in a single mortgage system at a large, full-service retail bank. But you have to consider, that over the years so many banks have been through mergers and acquisitions, they have had many competing systems, that it’s not uncommon for there to be many instance of such a system. I’ve heard of one bank with 50 different mortgage systems. And this is just mortgages, mix into that all the other activities that banks have, especially some of the complexities in investment banks, and all the risk systems, and you can imagine the scale of the architecture. I once saw an attempt to draw it out, each box was a couple of cm’s big, and it covered an entire wall or a pretty large meeting room. I know of an organization that is looking to modernize it’s architecture, it current has 8000 systems, being worked on by over 30k technology staff.
- With so many systems, so many changes being made, how can any centralized architecture function be reasonably certain of the impact of a change? How can security team understand the end to end flow, no one can, it’s impossible. But on top of that, with tens of thousands of technology staff actively developing, there thousands of change requests being made each year. I’ve personally witnessed the workload of some of these change review teams, they have a stack of requests in front of them. They can only reasonably spend a short amount of time on each one, they can’t the necessary time to perform detailed review. The same is true of operations teams, they are often not deeply familiar with the intricacies of the system, and are responsible for many systems. With dozens of systems to support, loads of changes to be implemented, and teams spread thinly, mistakes happen.
- So, with so many problems present, despite the efforts to improve quality and risk, how can we improve, how can to we solve these issues? Well, this is where I believe adopting Continuous Delivery can help.
- First off we should look to ensure we’re using modern version control such as Git (most places are these days), and implement a pull request based model, to enforce code reviews, this is an incredibly easy win and helps with code quality as well knowledge sharing. We also need to ensure we have a good build system, and binary artifact repository. We also should have code quality tools in place, checking for code complexity, bad practices, potential bugs etc, but also deviation from defined coding standards. I could talk more on this, on how for example google with 15k developers, and over 4000 active project, works on a single monolithic source tree, with all developers working on head, with sophisticated pre-tested commits. How it is build and tested (75 million test cases run each day) and all this happens in minutes. But we could take up an entire hour talking just about this. Instead, assuming most banks are not so sophisticated *yet*, but we already have a baseline of source code control, build infrastructure, unit tests and binary artifacts. Many Financial Service companies are already at this ‘baseline’ . The when can go on to look at the change process, when we actually stop to analyze much of the activity that goes on through these change review gates, we discover many formulaic process. In the case of the security reviews this is most certainly the case, penetration testing, packet sniffers checking network traffic, encryption checkers, cross site scripting checks, and so on. These are steps that can be automated. With the Security teams and Development teams working together to automated these processes, time will be freed in both teams, more security checks can be devised and we reach a point of being ‘very sure’ that every single release has no known security vulnerabilities. In Operations we can see many opportunities where Continuous Delivery can improve. There are some obvious wins as we look at automated/scripted deployments. However beyond that as we look to adopt Infrastructure as Code, we can not only improve the certainly that the environments we deploy to through the dev/sit/uat/prod cycle are consistent, we can reduce or eliminate the risk of fat finger deployment errors. We are also, very recently, at an incredibly exciting time when we look at contemporary software engineering. Specifically when we look at containers and when we look at infrastructure as code. If we adopt a service oriented architecture and more specifically build using micro services we are well positioned to reap several benefits. Firstly these more loosely coupled systems are more testable. We have cleanly defined interfaces, we can certainly more easily perform component, security and performance tests more easily on these services. But we can also look at consumer driven contracts, especially (but not exclusively) where we are able to build REST endpoints, consumers of a service can define test suites around the expectations of the service, and we can run these tests, we can even look at tools that use these contracts as simulations for a provided services. Finally, as we look at the container ecosystems, we find some incredibly exciting developments when we look at Mesos, Kubernetes, Docker suite. We are moving towards “Operating systems for the datacenter”. We are moving to a point where our operations team can stop worrying about ‘where’ to run services, but which versions to push out, how many should run, resource allocation etc. Working with operations we can build out automated deployments, and focus more on logs and metrics gathering, live monitoring, hot spot tuning - auto scaling and all round ‘run time’ improvements, rather than wasting time ‘managing releases’ Likewise as we look at data, we can certainly see some obvious testing that can be applied, for example scanning for CRUD operations vs stored procs, executing queries to ensure indices are being hit, testing performance etc. But again we can move up the maturity chain and look towards defining our data models in metadata repositories, defining data flows between systems. Collectively, all these steps help greatly when it comes to enterprises architecture. Again automated processes could scan for authorized software usage, but more than that if we focus less on static (and therefore immediately out of date) architecture documents, but rather auto generated documents, we can build a living breathing view of our enterprise, of the services, scale, capacity, performance, version. Of the data and the data flows within it. This is a powerful tool for architecture teams, working to rationalize, simplify, scale, etc.
- We can move to a model where we build highly automated production pipelines, the key to this is to Test All The Time * and Automate All The * Things
- I know of a number of banks that are adopting Continuous Delivery, but much of my experience is that these organizations shows, there are fundamental problems with adoption of these tools and techniques. They are inadvertently just paying lip service, they are hiring “DevOps engineers” to write build scripts, they are looking at virtualization and containerization technology, but they are struggling to affect the deep change in practice and the organizational transformation that is require to truly succeed in Continuous Delivery. This is not surprise as that is the way these silos are incentivized, it is the job of the development teams to get stuff done as quickly as possible for the business and are often under extreme delivery pressure, their default position is often ‘yes everything is safe and ready, let’s ship it’. It is the job of the change gatekeepers to ensure all risks are managed they are under extreme pressure to ensure no errors occur, and with such limited time their default position is often “no”. To truly succeed I believe we need deep organizational change and large shifts in the way we think and work. We need to move away from silos and into collaborations. We need to move away from change reviews that are like trials by fire.
- Financial Services IT often looks outwards to the broader software industry when looking at tools and technology, it should do similar when looking at processes and practices. Go study how Spotify, Google, Facebook, Amazon, Etsy etc work so effectively from a software delivery perspective. But also within the industry, there are isolated successes, at 51zero we have certainly had successes in being able to build big data solutions using Continuous Delivery success for a number of our banking clients. So find these isolated successes, study them and replicate them at scale.
- By changing the culture, by working highly collaboratively and achieving high levels of CD maturity we can move to High Velocity – Low Risk development
- There is one final risk I’d like to talk about, it is a big big risk that’s only just starting to be discussed seriously, the disruption risk. There are many dozens of FinTech companies popping up, they are lean, they do not have these horrific technology legacies, they are attracting top talent and they are moving fast. They are now starting to take money, hand over fist, from established firms. Without the agility that Continuous Delivery can bring the established firms face serious, very serious disruption risk.
- So I believe that the existing Financial Services industry has a choice to make, evolve or face extinction.
- Thank you