From nothing to production in 1 hour
•Download as PPTX, PDF•
0 likes•370 views
What does it mean for a big financial company to go large scale to the public cloud? What effect has this on the 200+ teams? What is needed to enable teams migrating their services from an on-premises modular monolith to a microservices architecture based on PCF, while ‘keeping the shop open’? We will share our lessons learned, how we enabled teams, how automation became our friend and what the costs are of full CI/CD. We will show what enables us to go from nothing to production within an hour. By: @_ht80_ and @rbraam
Recommended
More Related Content
What's hot
What's hot (20)
Similar to From nothing to production in 1 hour
Similar to From nothing to production in 1 hour (20)
More from Roy Braam
More from Roy Braam (7)
Recently uploaded
Recently uploaded (20)
From nothing to production in 1 hour
- 1. WelcomeClick to edit Master subtitle style From nothing to production within an hour Large scale @ Rabobank
- 3. Context
- 4. Linux box Portal Framework Shared libraries (API's) AppApp App App App App AppPlugins App IBM Websphere Application Server /* On-prem environment
- 5. 2008 - 2018 2 mln Logins / day 3000 Req. / second 150+ Teams 400+ Applications Very stable and controlled environment. Why do we need to move to a new one?
- 6. Move on Ageing technology Scalability limits DevOps teams
- 7. IBM Websphere Application Server Linux box Edge Service AppApp App App App App App App User Profile Page Render User Preference State Store Fraud Gateway /* Portal Framework Shared libraries (API's) Plugins
- 9. Simplicity MyApp Azure public cloud Pivotal Cloud Foundry Container Container MyAppMyApp Routingcf push myApp cf scale myApp -i 2
- 10. Cultural change and change is hard...
- 12. With freedom comes responsibility. Freedom makes a huge requirement of every human being. For the person who is unwilling to grow up, the person who does not want to carry his own weight, this is a frightening prospect. Eleanor Roosevelt
- 13. With freedom comes responsibility. Freedom makes a huge requirement of every DevOps team. For the team who is unwilling to grow up, the team who does not want to carry its own weight, this is a frightening prospect.
- 15. Some of our favourite experiences/quotes
- 16. “We need certificates to connect to these on- premises services, can you please send your certificates to us?“
- 17. “In my microservice I want to store a file in the container, how can I access it and serve it to a customer?”
- 18. “So, every team can deploy their own service to production. Why do we want that, it sounds very insecure! Why do we want this microservice / independent team thing?”
- 19. The first (big)disruptance in the Edge Service went unnoticed
- 20. • Cloud/PCF seminars • Microservice Guild • Weekly 'Walk-in cloud' • PCF enablement teams • Tooling for automation • Stackoverflow like Q&A • Chat rooms for Q&A • (opinionated) Documentation Help
- 21. EdgeService strategy Living on the Edge
- 22. domain : banking context : /api/balance userType : customer authLevel: 2-factor service : balance Edge Service Balance service User registry Edge service URL registry banking.rabobank.nl /api/balance authenticate dispatch balance.pcf-apps.rabobank.nl /health
- 24. User Authorization URL Governance Red Button Hiding endpoints CSRF Protection Circuit Breaking Cache headers Edge Service
- 25. “Why are you introducing a single point of failure?”
- 27. Prevent URL mess
- 30. CF in Azure On-premise PCF-productionTeam VM Concourse GIT team-space JIRA Infrastructure as code PR
- 32. Edge Service User Profile Customer name ROCS pipeline library Jenkins pipeline as code
- 33. ROCS pipeline library - Example
- 34. ROCS pipeline library - Example
- 35. https://martinfowler.com/bliki/BlueGreenDeployment.html - March 2010 Blue APP Green APP Blue APP Green APP CF prod in Azure Blue green deployment - zero downtime
- 36. Testing strategies How to test a microservice?
- 37. TDD with pair/mob programming Share knowledge Improves speed Improves quality Increases team spirit It’s fun Just try it out! Don’t do ADD!
- 38. #0 #0 #1 master develop feature x #2 #3 #4 #6 #7 #5 PR PR Testing strategies - our git flow
- 39. feature x Config Server Redis Stub Service A Stub Service B Edge service Testing strategies - feature branch
- 40. Work on a bunker without internet
- 41. develop Config Server Redis Stub Service A Stub Service B Edge service Testing strategies - develop branch
- 42. master Stub Service A Stub Service B Customer name Balance service Edge service Testing strategies - master branch
- 43. “When do you actually integrate with other microservices / backends?”
- 44. Feature toggle Verification route Nothing looks more like Production then Production We do it in production Monitoring Metrics
- 45. Enterprise wide testing Every time you do it… a small puppy dies at Madurodam! :(:(:(
- 46. CF in AzureCF in Azure Active - Active Extreme testing - Chaos Harry
- 47. PCF in the public cloud Edge Service CI/CD Recap
- 48. Next challenges What is next
- 49. Next... Page rendering A/B Testing Canary releases Service-mesh with Istio
- 50. Questions?
Editor's Notes
- GOOD MORNING EVERYONE How many of you have worked in a Bank? Production and bank regulation - Public cloud - Old on-prem env. takes 4 ours with a lot of automation - Here we are, multiple times a day, 10 minutes - Demonstrated internal conf. 1 hour production - Looking back, what, bank/tech - Thanks, lots of quality talks to choose
- From, what, to
- On prem, nothing special Own portal impl Modular monolith Our team responsible for orange blocks
- - 10y served us well - Almost no disruptance
- Aging Technology: Topics during conferences. Could not choose fitting technologie Scaling: Lots of effort to scale PCF: Per microservice and on platform level DevOps: Not only say we are doing devops Deploy independent to production. When we deploy no other team can do it
- From this, to: This
- Explain EdgeService later Orange is ROCS On the ROCS, shaken not Stirred (dr. No 1962)
- Let the platform run it Focus on building functionality Fintan Ryan: Senior research Director and analyst new platform ~250 teams. (10 min)
- Not only tech change, also cult.
- What can go wrong
- We use this quote a lot Autobiograpy "you learn by living(1960)" Devops teams
- What is cost of freedom, moving away controlled env.
- And a lot more...
- Don't take this personal if your qoute is in here Not going to say which ones are my own...
- It's like passport border control
- Stateless Instances, rolling upgrades.
- Security intake.
- Restarting instances, no config server available. In 2 foundations. Silence
- (15 minutes)
- Common Edgeservice strategy
- From outside not able to connect to health endpoints Some intake on urls, automate.
- Takes less than 15 minutes to update
- User Authorization: Check if the (logged in) user is allowed to access a service with the requested HTTP methods Governance: multiple departments, multiple business domains, but the same web domain. Where we want governance on the exposed API's Red button: An emergengy button to disable inbound traffic to a certain service and minimize damage. Circuit Breaking per route. Breathing space and fail fast. Control over the endpoints to expose Setting default headers when not set, for example caching. Rate limit
- You might ask
- - Because developers are human and might forget something I know, everybody in this room never makes mistakes, but there is also a world outside this room
- - Everybody thinks about URL naming right? - Nice, full rest urls, with resource names and HTTP Methods etc. - Separation between URL design and implementation - Good rules cross domains. - Governance on the URLs we expose as the domain of the bank
- - Circuitbreaker to allow broken services to recover - Rate limit Stop traffic to service when disfunctions (red-button) (20 minutes) This reduces time to market along with automation
- We can only go to production within one hour if we abuse automation
- Panzer team is only 3 people to manage all the pcf environments! Less then 15 m later we have access
- Duplication initially and we knew we had to remove it later Then we created ROCS pipeline library!!!! Rabobank Online Cloud Services Or Core as some might call it… Community driven library
- Each change in production needs to be registered in our ticketing system (SM9) and that is mandatory. We took a lot of effort to automate this part. Withoutdown time will be explained in the next slide
- Verification route is available in a production environment All interactions are made through Jenkins pipeline using a functional account With this we can deploy fast to production, but what about the quality of what we deliver?
- Testing will allow us to increase quality (30 minutes)
- Try it out and get into the TDD mindset ADD is Assumption driven development, Don’t assume your backend is always there or it always returns a 200. Test your failure scenarios.
- Why we have two PR’s? I will explain in the next slides
- - Stub all dependencies - Test in ISOLATION - No dependency on infrastructure or network
- Try to integrate as much as possible in production Don’t do enterprise wide testing! With enterprise wide testing we mean running your tests agains a production like environment
- Draw two pcf foundations and how we stop one region and things should still continue working We do this quite often and everything runs fine but sometimes applications fail 50% or never work since they deployed them manually
- From nothing to production in one hour is possible due to PCF in the public cloud - 15m Edge Service to be able to expose our features to the customers through an automated process - 15m DONE ONCE After the initial set up you only need the CI/CD that takes us 10minutes
- - Mention Service Mesh - ISTIO?
- Questions?