Talk by Mitesh Sharma, Technology Lead at Kiwi on the topic "Cost optimized logging using API Gateway, SQS and Elastic Search" at AWS Community Day, Bangalore 2018

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cost Optimized Logging Using API
Gateway, SQS and ElasticSearch

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Mitesh Sharma
Tech Lead at Kiwi India Pvt Ltd
Find Me:
LinkedIn : https://www.linkedin.com/in/mitesh-sharma-76722015
Twitter : https://twitter.com/mitesh_dev
Github : https://github.com/MiteshSharma
Who Am I?

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What We Do?

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Joyride App
Play Social Real Time Interactive Game Shows
Players Can Win Real Cash Prizes

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Game shows are fun to play
Social interaction in real time games
Players can play in team or solo
Players win real cash
During game shows, multiple messages exchanges in real time
In last 1 minute of game show start, everyone enters app

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What Problem We Want To
Solve?

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Identify customer issues during game show,
fix them and give them best experience

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How to solve customer issues?

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Capture logs, understand them, identify
issues, pinpoint code where things went
wrong and fix them

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Client App
Logs on
server
Investigate

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Solution should be robust, cost effective and
scalable

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What we build?

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Route53
Cloudfront
Log
Service
MongoDB
EBSELB

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How to scale as traffic
increases?

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Auto Scaling group helps with cost and handle huge traffic

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Auto Scaling group helps with cost and handle huge traffic
All users come in less than a minute, so auto scaling can’t help

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Auto Scaling group helps with cost and handle huge traffic
All users come in less than a minute, so auto scaling can’t help
Use time based instances

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Auto Scaling group helps with cost and handle huge traffic
All users come in less than a minute, so auto scaling can’t help
Use time based instances
Once game show ends, traffic becomes normal
Multiple game shows scheduled one after another
Effectively need to keep all instances running for whole day
Cost of running multiple instances are high

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What other components were
taking up cost?

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
● Self hosted MongoDB cluster
● MongoDB needed high IOPS EBS to write these logs as
they arrive
● Used Provisioned IOPS EBS
● When there is less traffic, still paying for high provisioned
IOPS
● Needed to write queries for fetching data from MongoDB
● Build custom dashboard for logs

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What can go wrong here?

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Route53
Cloudfront
Log
Service
MongoDB
EBSELB
Crash
Crash

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How we solved these problems?

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Route53
Cloudfront API Gateway SQS ElasticSearchFetcher
Kibana

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why we choose each service?

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS API Gateway

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
● Easy to manage
● Easy to scale
● No minimum fees
● Act as Proxy
● Endpoints are public
● Easy monitoring using cloudwatch
● Perfect for our use case

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Route53
Cloudfront API Gateway

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS SQS

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
● Messaging queue service
● Reliable and scalable
● No minimum fees
● Provides decoupling
● Resilience
● Push/Pull architecture
● At-least-once delivery guarantee
● Easy monitoring using cloudwatch
● Receive messages from API Gateway for later processing

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Route53
Cloudfront API Gateway SQS

32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS ElasticSearch With Kibana

33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
● Real time, distributed, search and analytics engine
● Schema free
● Developer friendly REST APIs
● Easy to deploy and manage cluster
● Secure
● Comes with build-in Kibana
● Kibana provides visualization of data in elasticsearch

34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Route53
Cloudfront API Gateway SQS ElasticSearch
Kibana

35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Fetcher Service

36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
● Custom java service
● Periodically polling
● Send messages to Elasticsearch
● Dynamic polling behaviour
● Delete messages once processed

37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Route53
Cloudfront API Gateway SQS ElasticSearchFetcher
Kibana

38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What can go wrong here?

39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Route53
Cloudfront API Gateway SQS ElasticSearchFetcher
Kibana
Crash

40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Learning

41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
● Use managed services and focus on solving problem
● SQS can be really good in making asynchronous architecture
● Add alerts on SQS message duration
● Push messages in bulk in elasticsearch
● Keep deleting old messages from elasticsearch
● Cost is part of your efficient architecture
● Making customer happy is not easy

42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thanks