Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Celebrate Failure


Published on

How recognizing and using opportunities makes it possible to implement changes quickly and without loss of service in a 7x24 hour online world. Even if not everything goes according to plan.

Published in: Economy & Finance
  • Be the first to comment

  • Be the first to like this

Celebrate Failure

  1. 1. Celebrate Failure Angelique van den Hoeven, Jeroen Wallaard, Mikel van Son Getting It Done Leipzig, May 16th 2018
  2. 2. Introducing ING 2 ING’s strengths include a well-known brand that is positively recognized in many markets, a strong financial position, our international network and omnichannel distribution strategy. Not to mention our efforts to contribute to a sustainable world. ING is among the leading banks in the Dow Jones Sustainability Index (Europe and World) and is included in the FTSE4Good index. Who are we? ING NL • > 11 million Current Accounts • 4.2 million logons in the Mobile App per day • > 13 million money transactions per day • > 27 million balance inquiries per day Key facts
  3. 3. RAK the critical banking application with two main capabilities 3
  4. 4. During capacity analysis we found that MQ could become a bottleneck in the near future 4 Now vs Forecast batch processing - Normally 15000 tps incoming Credit transactions - Forecasts 25000 tps incoming Credit transactions Now vs Forecast online transaction load - Normally 7000 tps online transactions - Forecasts 14000 tps online transactions Online transaction load Batch processing No balance information available No transaction information available The impact of a system crash MQ APP
  5. 5. First upcoming peak-day was next Payday. Meaning we had one challenge: time 5 Weekend WeekendWeekend DR Failback DR PayDay Day -5 Day -2 Day +5 Day +13 New Idea Day 1 Activate Quad- Core on DR Day +3 No Acceptance Environment Test rebalanced MQ Day +7 Deploy rebalanced MQ Day +9 Activate Quad- Core on DR Day +6
  6. 6. On Day 1 we had an idea to switch back to a 4 core enabled server 6 • 08:23, Received remaining licenses from HPE • 09:30, Activated the new license files and the HPE Core License and we went to the stand-up • 09:40, minutes later CPU 7 Halted (due to a bad core) • 16:30, CPU 7 replaced • Wait for permission to cold-load the DR system • 17:45, Cold-loaded the serve to forcefully rebalance all processes • 18:00, finished Unlucky Day • During the day received remaining agreements and necessary license files (non-HPE) • Implementation manual finished • HPE account team performed their magic WhatsApp (again ) “Jeroen, can we perform the MQ rebalance coming Monday. Can you arrange that? Pieces coming together • Set the deadline for core activation on Friday 14:00 • Created necessary changes • Asked our suppliers for license and commercial impact • In this case it helped that most of the suppliers are in different time zones • First agreements were reached and license files (non-HPE) were received overnight Deadline “17:17 Jeroen, we would like to switch back to a 4 core enabled server. Can you arrange that? Mikel, could be a commercial challenge. Jeroen, to be honest not my problem. Luckily we already started a week before with looking into the options / consequences of turning on the extra cores WhatsApp Day 1 Day +1 Day +2 Day +3
  7. 7. Activating the Quad-Core on the regular DR system was flawless, including cold load and sanity testing on the Acceptance environment. Preps for the tests on Day +7 were made. Activating the Quad-Core on the regular DR system was flawless. However, the roller coaster ride continued after Day +3. Namely, testing the solution 7 Weekend WeekendWeekend DR Failback DR PayDay New Idea Activate Quad- Core on DR No Acceptance Environment Test rebalanced MQ Deploy rebalanced MQ Activate Quad- Core on DR We are here Day -5 Day -2 Day +5 Day +13Day 1 Day +3 Day +7 Day +9Day +6
  8. 8. We had a successful MQ rebalance testing day on Day +7 to simulate our forecasted peakload 8 Day +7: Timeline of MQ rebalance testing day 09:00 12:00 17:00 21:00 Started with a highly motivated team of Infra- and Application engineers Lunch was arranged upfront Planned end-of-day Pizza time! We ended our first successful test-run
  9. 9. 9 days after the idea we successfully deployed the new solution 9 02 04 Success factors 03 01 Solid plan on what to do Dedicated team Agile organization Smooth transformation
  10. 10. Questions? 10 I’m user- friendly