Download to read offline
Uploaded byTomas Riha
Into the cloud
The document outlines the transformation of a continuous delivery infrastructure at WirelessCar, led by Tomas Riha, focusing on migrating to a microservices architecture using AWS. Key achievements include automated provisioning, reduced maintenance costs, and scalability enhancements, enabling over 400 developers to utilize the platform effectively. The migration process involved strategic planning, cross-organizational collaboration, and significant changes to operational responsibilities to streamline the development pipeline.
Into the cloud
- 1. Into the Cloud Makingour Continuous Delivery fly.
- 2. Tomas Riha Architect @VGT/WirelessCar Passionate about creativity, change and improvement Horrible at following instructions and performing repetitive tasks MAJOR Project Liability mail: tomas.riha@wirelesscar.com twitter: @TomasRihaSE blog: continuous-delivery-and-more.blogspot.com
- 3. Four Years ago NewTelematics Delivery Platform Sent us on road to Micro Services Continuous Delivery
- 4. Infrastructure at thattime Low level of virtualization “Two servers and a Oracle DB” per delivery Split Operations and Development Infrastructure and Application Architecture separated Long process to get a server
- 5.
- 6.
- 7. Continuous Delivery Build ReleaseDeploy Test Build Release Deploy Test Build Release Deploy Test Build Release Deploy Test Build Release Deploy Test Assemble Release Deploy Test Service Pipes 100s of tests run in parallel testing Solution Pipes 10s Use Case Requirements Verified
- 8. Here is yourtest server Build Release Deploy Test DB Test Server
- 9. Static virtual instances MasterWorker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Custom DeliveryEngine GUI Master Mind Gerrit
- 10. Capacity Planning &Peaks Master Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker ?
- 11. Pampering the Snowflakes Eachstatic server needs maintenance. Server Maintenance one of the top task the team does Each server provisioned by different version of our chef scripts, where baseline is a moving target
- 12. Continuous Delivery asa Service Ok so this Continuous Delivery thing is nice.... ....what does it take to deliver it to the entire organization as a Service? 400+ developers
- 13. Continuous Delivery asa Service Blue skies, Sunshine and ohh yes a CLOUD PROVIDER!! Amazon Web Services
- 14. Load Tests We haveused AWS a bit for Load Tests
- 15. Go! Continuous Delivery inthe AWS Cloud Demo Environments in the AWS Cloud
- 16. Build in Cloudhost on premises and at partner AWS Volvo Hosting Partner Hosting Partner Compatibility in Environments!
- 17. What we got Automatedprovisioning on our Continuous Delivery infrastructure using chef Automated provisioning of our Jenkins jobs into build pipes Rudimentary Image Baking Capabilities from our AWS Load Tests Basic understanding of how our services behave in a cloud environment
- 18. What we needto do Build a Virtual Private Cloud with networks Connect VPC to Volvo Network Migrate about ~800 Jenkins jobs 60 Jenkins master/slaves 1 Nexus server, 2 test databases, 1 Sonar, 1 Elk 2 custom applications, 1 MongoDB
- 19. Our approach Change aslittle as possible during migration Automate everything Establish responsibility between Operations and Development
- 20.
- 21. Defining Application Topology LoadBalancer Server Instance Server Instance Data Storage In Amazon Web Services ● Elastic Load Balancer ○ Security Groups ● Auto Scaling Group ○ Launch Configuration ○ Amazon Machine Image ○ Security Groups ● Data Storages ○ Simple Storage Service ○ Relational Data Service ■ Oracle ■ Postgres ○ Custom Databases ■ MongoDB ○ Security Groups
- 22. Defining Application Environment LoadBalancer Server Instance Server Instance Data Storage In Amazon Web Services ● Virtual Private Cloud ○ Subnets ● Auto Scaling Group ○ Scaling Rules ■ Up/Down ■ Max/min ○ Instance Size ■ Mem/Cpu ● Data Storages ○ Size ■ Mem/Cpu/Storage
- 23. Infrastructure as Code DevTest Load Test UAT INT Test Provisioning Interface Vagrant Impl Cloud Provider Impl PDF Document Impl Topology Spec Env Spec Legacy PROD Environment Definition Networking Loadbalancing Servers Databases Location Integration Scaling
- 24. Infrastructure as Code Test Load Test ProvisioningInterface AWS Cloudformation Impl Topology Spec Env Spec Demo Delivery Engine Tool
- 25. Baking Machine Images GitRepo Base Packer Scripts Bakery Instance Git Repo Delta Packer Scripts AMI Alpha RHEL AMI Bakery Instance AMI Delta
- 26. Baking Machine Images RHEL AMI VGTBase Java Base Jenkins Base Nexus Graphite ELK Jenkins Master Jenkins SlaveApplications Applications Applications
- 27. Chef Issues Never sharesame scripts for baking image that go from known state to a delta with scripts that manage snowflakes!
- 28. Phase 1 Politics, CrossOrganizational handovers & PoC Way too much time Phase 2 Build & Automate Setup Build & Tear Up the VPC a few 100 times 1.5 months ~ 2 engineers Phase 3 Migration Start moving over the repositories. Add .delivery-engine.yml files to repositories Disable build pipes in old environment 1 month ~ 3 engineers The Migration
- 29. Daily Cost trackingduring migration!! Did we do something really stupid yesterday? Bosses love you when you report cost and think about optimizing cost Cost Tracking
- 30.
- 31. Since the migrationstarted we have not launched one single instance through the AWS Console. Console only used to terminate instances, never to create instances. Done well
- 32. Delivery Engine afterMigration More or less the same setup Different distribution of jobs Auto Scaling Workers Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Custom DeliveryEngine GUI Master Mind Gerrit
- 33. Immediate Benefits Maintenance downat least 80% at least due to worker recycling No more capacity planning Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Custom DeliveryEngine GUI Master Mind Gerrit
- 34. Not High Availability Setupis still stateful Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Custom DeliveryEngine GUI Master Mind Gerrit
- 35. 250 Git repositoriesbuilding, testing, releasing Components, Applications, Solutions For closer to 150 developers Usage
- 36. Growth during migrationwasn't expected but was about 50 repos Over 100 Git repos added after migration New teams constantly onboarding to Delivery Engine Expecting 200 plus developers by end of year Growth
- 37. Grow number ofusers and content not the size of our Team Size! We do it through a Event Driven, Stateless, Application developed using ALL the Continuous Delivery principles. Using Continuous Delivery to Deliver Continuous Delivery. We need to ensure scalability and availability
- 38. Stateless Build Environment Pipesas Code Unit test pipe task Integration test pipes Job Pipe as Code Client
- 39. Stateless Build Environment BuildData and reports away from Build Environment Job Pipe as Code Client
- 40. Stateless Build Environment NoDependency between GIT Repository and Build Job Only one job exists Job Pipe as Code Client Queue Gerrit Listener
- 41. Job Stateless Build Environment AwsTools Pipe as Code Client Test Environment Queue Gerrit Listener Split Test Environment from Build Environment
- 42. HA Build environment TestEnvironment Rabbit MQ Cluster Coordinator Jenkins Factory Foreman Jenkins Workers Jenkins SlavesWorkers Jenkins SlavesWorkers Jenkins Slaves Gerrit Factory Foreman Jenkins Workers Jenkins SlavesWorkers Jenkins SlavesWorkers Jenkins Slaves Test Environment Test Environment Test Environment ELK Graphite S3 Master Mind Custom DeliveryEngine GUI Slack
- 43. From idea torolled out in about 2 months, 1 man month of work Without AWS not possible HA Rewrite
- 44. + Lead time forInfrastructural change drastically reduced Ability to build HA and scalability on a level previously not possible No more capacity planning Demo Environment using same tooling and fully up and running QA and Integration testing environments coming up as well DevOps Amazon Web Services works GREAT! - Different tooling for creating aws infrastructure to creating in house infrastructure Retrospective
- 45. Questions? here: @TheBar mail: tomas.riha@wirelesscar.com twitter:@TomasRihaSE blog: continuous-delivery-and-more.blogspot.com Thats it!