Downloaded 11 times
Uploaded byAWS User Group Bengaluru
Lessons learnt in CI/CD with AWS serverless architecture
The document outlines lessons learned in Continuous Integration/Continuous Deployment (CI/CD) using AWS serverless architecture, emphasizing the transition from Jenkins to AWS services due to challenges with cost, maintenance, and security. It details the needs and processes implemented for streamlined deployments, including integration tests and infrastructure management. Key takeaways include best practices for unit testing, security monitoring, and cost management in the development pipeline.
More Related Content
Similar to Lessons learnt in CI/CD with AWS serverless architecture
More from AWS User Group Bengaluru
Lessons learnt in CI/CD with AWS serverless architecture
- 1. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. BENGALURU
- 2. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Lessons Learnt in CI/CD with AWS Serverless Architecture Sriguru V | 06-Oct-2018
- 3. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Agenda Background Needs How we started ? Challenges Lessons Learnt Key Takeaways
- 4. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Background AngularJS Front-End Application AWS Serverless Architecture Connected Products
- 5. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Needs identified Developer • Speed in deployments • Integrated testing of Unit Tests, E2E Tests, API Tests • Minimal time of Build, Test and deployment process • Replicable infrastructure for pipelines • Different environments
- 6. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Developers How we started ? Developer 1 2 3 Push/Pull/Commit Jenkins Server Build triggered through Web Hooks Publish to S3 on Successful Build
- 7. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Challenges with jenkins adoption Developer • Cost • Operation issues • Dependencies on the Plug - ins • Unnecessary heaving lifting • Maintenance of security of the instance
- 8. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Different Environments Developer Development QA Pre Production Production Build Test Release Build Test Release Build Test Release Build Test Release Our Customer Verified / Validated by Dev. Team Verified / Validated by QA. Team Verified / Validated by Customer Plan Monitor Development Team
- 9. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Developers Jenkins replaced with AWS Services Developer 1 2 Push/Pull/Commit Build triggered through Web Hooks CodeBuild Deployment Cache Invalidation 3
- 10. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Tools adopted Developer SCM Unit Test Build E2E Deploy& Invalidation
- 11. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. AWS Services – Post adoption challenges Developer • Cross account issues in the pipelines • Adoption appropriate build tools • Customized docker container in Code Build for manage End to End Tests • Integration with Agile tools • Email and Slack Notifications • Bloated containers to be skimmed for only necessary libs
- 12. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Cross Account Handling, Pipeline Steps Developer buildspec_ut.yml buildspec_e2e.yml Unit Testing E2E Testing S3 Bucket Invalidate Halt Operation QMetry JUnit Reporter CI CD Jira Tickets Developer Account Production Account Personal Access Token 1 2 3 4 5
- 13. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. SQS and SES Integration CloudWatch Rule Cloud Formation Template – Infrastructure as a Code Managing the pipeline
- 14. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Risks and Mitigation Plan Version Issues in npm Bower to yarn migration E2E Integration
- 15. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved. Best Practices Checklists adoption Unit Testing of scripts Keep track of Domains and Route53 Continuous monitoring of Security Advisor Update the CLI commands on Version upgrades in Code Build Frequently review budget through cost explorer
- 16. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved.
- 17. © 2018, AmazonWeb Services, Inc. or its Affiliates. All rights reserved.
Editor's Notes
- #5 This is the overview of the solution context we are going to discuss on today Solution has been developed for a North American OEM who develops High end products for Education, Government and Corporate customers We have a front end application which helps the end users of multiple tenants and OEM to work with connected products We have adopted serverless approach and this session is going to talk about the various lessons learnt in the implementation of this solution Development was done using agile practices
- #6 Speed in deployments Integrated testing of Unit Tests, E2E Tests, API Tests Minimal time of Build, Test and deployment process Ability to recreate the pipeline infrastructure
- #7 We first setup of dev environment with Dev Branch Adopted Jenkins with plugins for managing the Build and deploy jobs Basically using the Web hooks to trigger the unit tests and then build process and subsequently Upon successful build deploy to S3 Bucket
- #8 We had some security threats and cost impact of the running instances Customer is interested in going more serverless to understand granularity of cost by leveraging AWS services Security of EC2 instances Some of the team members has to maintain the EC2 instance for the Jenkins jobs which also to be monitored and managed Upgrade plug-ins, etc.,
- #9 After identify some of the challenges as outlined in the previous slide on Jenkins we restored to use AWS services We were getting into the need for different environments for different stakeholders Dev - Dev Team QA – testing Team Pre Production – UAT/Business Team Production – Customers This also defined our approach on code deploy and manual handoffs required for approval
- #10 We have replaced Jenkins with AWS services The docker containers will be running in the code pipeline Separate credentials with IAM polices for dealing with Code pipeline execution Code Build Services would be adopted for Build & Tests, Deployment and Cache Invalidation to complete the deployment This would get replicated for different environments
- #11 Serverless code, API Test Scripts, Unit Test Scripts, E2E Test Scripts, Database migration scripts are stored in the GitHub repository Test libraries and its dependencies (Example: npm) are managed to be more lightweight through custom built docker containers Used tools like Gulp/Yarn for managing dependencies and deploy: Obfuscation Cache Invalidation (CloudFront) Deployment Caching
- #12 Code build process should handle obfuscation, minification of the front end code Headless chrome adoption for running the End to Tests Integration of test results for Slack Notifications Integrate the Qmetry test results with AWS Pipeline through S3 Buckets Managing the pipelines for managing cross account issues between dev and production