Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cost optimized logging using API Gateway, SQS and Elastic Search

31 views

Published on

Talk by Mitesh Sharma, Technology Lead at Kiwi on the topic "Cost optimized logging using API Gateway, SQS and Elastic Search" at AWS Community Day, Bangalore 2018

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Cost optimized logging using API Gateway, SQS and Elastic Search

  1. 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cost Optimized Logging Using API Gateway, SQS and ElasticSearch
  2. 2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Mitesh Sharma Tech Lead at Kiwi India Pvt Ltd Find Me: LinkedIn : https://www.linkedin.com/in/mitesh-sharma-76722015 Twitter : https://twitter.com/mitesh_dev Github : https://github.com/MiteshSharma Who Am I?
  3. 3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What We Do?
  4. 4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Joyride App Play Social Real Time Interactive Game Shows Players Can Win Real Cash Prizes
  5. 5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Game shows are fun to play Social interaction in real time games Players can play in team or solo Players win real cash During game shows, multiple messages exchanges in real time In last 1 minute of game show start, everyone enters app
  6. 6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What Problem We Want To Solve?
  7. 7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Identify customer issues during game show, fix them and give them best experience
  8. 8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How to solve customer issues?
  9. 9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Capture logs, understand them, identify issues, pinpoint code where things went wrong and fix them
  10. 10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Client App Logs on server Investigate
  11. 11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Solution should be robust, cost effective and scalable
  12. 12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What we build?
  13. 13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Route53 Cloudfront Log Service MongoDB EBSELB
  14. 14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How to scale as traffic increases?
  15. 15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Auto Scaling group helps with cost and handle huge traffic
  16. 16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Auto Scaling group helps with cost and handle huge traffic All users come in less than a minute, so auto scaling can’t help
  17. 17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Auto Scaling group helps with cost and handle huge traffic All users come in less than a minute, so auto scaling can’t help Use time based instances
  18. 18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Auto Scaling group helps with cost and handle huge traffic All users come in less than a minute, so auto scaling can’t help Use time based instances Once game show ends, traffic becomes normal Multiple game shows scheduled one after another Effectively need to keep all instances running for whole day Cost of running multiple instances are high
  19. 19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What other components were taking up cost?
  20. 20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ● Self hosted MongoDB cluster ● MongoDB needed high IOPS EBS to write these logs as they arrive ● Used Provisioned IOPS EBS ● When there is less traffic, still paying for high provisioned IOPS ● Needed to write queries for fetching data from MongoDB ● Build custom dashboard for logs
  21. 21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What can go wrong here?
  22. 22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Route53 Cloudfront Log Service MongoDB EBSELB Crash Crash
  23. 23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How we solved these problems?
  24. 24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Route53 Cloudfront API Gateway SQS ElasticSearchFetcher Kibana
  25. 25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Why we choose each service?
  26. 26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS API Gateway
  27. 27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ● Easy to manage ● Easy to scale ● No minimum fees ● Act as Proxy ● Endpoints are public ● Easy monitoring using cloudwatch ● Perfect for our use case
  28. 28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Route53 Cloudfront API Gateway
  29. 29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS SQS
  30. 30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ● Messaging queue service ● Reliable and scalable ● No minimum fees ● Provides decoupling ● Resilience ● Push/Pull architecture ● At-least-once delivery guarantee ● Easy monitoring using cloudwatch ● Receive messages from API Gateway for later processing
  31. 31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Route53 Cloudfront API Gateway SQS
  32. 32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS ElasticSearch With Kibana
  33. 33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ● Real time, distributed, search and analytics engine ● Schema free ● Developer friendly REST APIs ● Easy to deploy and manage cluster ● Secure ● Comes with build-in Kibana ● Kibana provides visualization of data in elasticsearch
  34. 34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Route53 Cloudfront API Gateway SQS ElasticSearch Kibana
  35. 35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Fetcher Service
  36. 36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ● Custom java service ● Periodically polling ● Send messages to Elasticsearch ● Dynamic polling behaviour ● Delete messages once processed
  37. 37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Route53 Cloudfront API Gateway SQS ElasticSearchFetcher Kibana
  38. 38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What can go wrong here?
  39. 39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Route53 Cloudfront API Gateway SQS ElasticSearchFetcher Kibana Crash
  40. 40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Learning
  41. 41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ● Use managed services and focus on solving problem ● SQS can be really good in making asynchronous architecture ● Add alerts on SQS message duration ● Push messages in bulk in elasticsearch ● Keep deleting old messages from elasticsearch ● Cost is part of your efficient architecture ● Making customer happy is not easy
  42. 42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thanks

×