Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Efficient, Scalable and Resilient Front-end logging service with AWS


Published on

Building Efficient, Scalable and Resilient Front-end logging service with AWS conducted at AWS Community Day, Bangalore 2019

Published in: Technology
  • Be the first to comment

Building Efficient, Scalable and Resilient Front-end logging service with AWS

  1. 1. Building Efficient, Scalable, and Resilient Front-end Logging Service with AWS KOKILAVANI KATHIRESAN | 27/07/2019
  2. 2. Introduction - Containers, Serverless, Microservice Architecture change the way the software is built - The systems are more distributed, and more ephemeral - No Complex system is ever fully healthy - Better Resilience and Fault Tolerance is the goal - Ease of debugging is a cornerstone to maintain and evolve robust systems
  3. 3. Observability - Internal states of the system should be inferred by its external outputs - Reduce MTTD and MTTR - Verifying the health of the service proactively - To know what’s broken, and why? - Provides the all-important feedback that drives future iterations
  4. 4. Our Business Case - To Collect logs, traces and metrics from Mobile/Web Browser - Get insights of the application - Understanding the user behavior patterns - Monitor application performance
  5. 5. Front-end Logging Service - Exposed a REST Endpoint - Spring boot application which accepts the compressed log message - Decompress and Validate the Payload - Forward it to the application’s log destination (Splunk) Requirements: - 20000 Transactions per second - 1 second latency Internet Logging Service AWS Account Compressed Batched Logs
  6. 6. Latency Improvement We split the service into two microservices. Producer: - Receives request and Validate the sender - Accepts the payload - Puts the data to queue Consumer: - Polls the data from queue - Extract the payload and Validate the data - Sends it to log destination Logging Service - Producer Logging Service - Consumer SQS
  7. 7. FE Architecture in AWS SHAILJA AGARWALA
  8. 8. Well Architected Framework Five pillars : - Operational excellence - Security - Reliability - Performance efficiency - Cost optimization
  9. 9. EC2 Setup Producer: - Compute Intensive (c5.2xlarge) - No of instances : 3 to 20 Consumer: - Memory Intensive (m5.2xlarge) - No of Instances : 3 to 20 Alarms: - Based on JVM metrics sent to Cloud watch
  10. 10. Load Balancer EC2 EC2 EC2 EC2 EC2 EC2 SQS ELB ELB
  11. 11. Route 53 - Expose the producer ELB through Route 53 - Route 53 endpoint is hosted behind Intuit API gateway - Disaster recovery through multiple CName across region EC2 EC2 EC2
  12. 12. Route 53 config
  13. 13. Route 53 config contd.
  14. 14. Auto Scaling Group Log generated varies during tax peak across the year Producer: - Request Processing Time decides scaling Consumer: - SQS depth
  15. 15. Auto Scaling Policies
  16. 16. Target Groups - With auto scaling and load balancers involved, target groups will route requests to EC2s and microservices - Requests are being sent to new targets as soon as the registration is complete and initial health check is passed
  17. 17. Cloud formation – Infrastructure as Code
  18. 18. Deployment & AMI Restack RAVIKUMAR KOTTA
  19. 19. AMI Restack Background: - Intuit compliance team applies security patches and new baseline images are released every 2 weeks - App teams must either use these AMIs or derive AMIs from those baseline images - Automated this entire process by using CW Rule and Codebuild services
  20. 20. Config: CW Rule on rhel7.4
  21. 21. Code build logs - Baking Logging service AMI - Launch the new EC2 instance from Baseline AMI - Copy chef recipes required to install software like java etc.. and configuration required for Splunk forwarder and log rotation - Bake logging service AMI - Publish cloud watch event with the AMI id
  22. 22. Code build logs - Baking Logging service AMI
  23. 23. CW rule on Baked AMI - Cloud watch rule configured to trigger on baked logging service AMI - We have 2 targets configured on this CW Rule - Lambda function: Creates new launch config with new AMI and updates ASG - Code pipeline: CD service to automate the steps to release logging service
  24. 24. CW rule on Baked AMI
  25. 25. Code Pipeline to automate Deployment process. - Source Stage: Downloads app config files - Code deployment stage: Reads app file from source stage and triggers code deploy for all environments - Blue-Green deployment - Re-routing traffic to new instances
  26. 26. Deployment
  27. 27. Lifecycle hooks - BeforeInstall: Setup application configuration for ex: install jre, collectd, splunk forwarder and pulls the latest code from S3 and deploy it - ApplicationStop: Stops application - ApplicationStart: Starts application - ValidateService: Invokes automation tests against deployed code(Green) App Spec Config
  28. 28. Installing Application and dependencies
  29. 29. Performance Test Report
  30. 30. Title + Content Golden Signal Metrics
  31. 31. Enhancements - Extension for Metrics and Traces - Dockerize the service code - Deploy in Kubernetes
  32. 32. Thank you!