This document compares two designs for visualizing S3 web page log data: Athena-QuickSight and Elasticsearch-Kibana. The Athena-QuickSight design involves transforming logs into a format QuickSight can read, storing them in S3, and using Athena to query the data and create visualizations in QuickSight. The Elasticsearch-Kibana design transforms logs to JSON, stores them in Elasticsearch, and allows Kibana to query Elasticsearch and create visualizations. Both designs are evaluated based on capabilities, costs, and security considerations. QuickSight is generally easier to use but more limited than Kibana. The Elasticsearch-Kibana design has higher monthly operating costs but provides more

1. ©Practical Code, LLC 2019
Visualizing S3 Web Page Log Data
Comparison of QuickSight and Kibana
Syenna Graham
Systems Engineer
02/01/2019

2. ©Practical Code, LLC 2019
Practical Code LLC
Our Mission: To Enable Those Soling the World’s Hardest Problems
“There is an elegant solution to every problem.”
“Unmanageable esoteric code has twice the cost of maintainable
code.”
“Automation is achieved by addressing one workflow at a time.”
COMPANY INTRODUCTION

3. ©Practical Code, LLC 2019
Design Considerations

4. ©Practical Code, LLC 2019
Comparison of Designs
Athena-QuickSight
Elasticsearch-Kibana

5. ©Practical Code, LLC 2019
Athena-QuickSight Design
Walk-through of design
1. The S3 bucket contains logs for the web page
2. When a new log is added to the S3 bucket, the Lambda will transform the log into a form
QuickSight can read and place them into a new S3 bucket
3. The Lambda function can be used to process the backlog
4. QuickSight can now ask Athena for a query to create visualizations
5. Athena will query the S3 bucket with it’s pre-set schema

6. ©Practical Code, LLC 2019
Elasticsearch-Kibana Design
Walk-through of design
1. The S3 bucket contains logs for the web page
2. When a new log is added to the S3 bucket, Lambda will transform the log into a JSON file and
post it to Elasticsearch
3. Elasticsearch will store the file in its database
4. Kibana can query Elasticsearch to make visualizations of the logs
5. Cognito will control access to Kibana

7. ©Practical Code, LLC 2019
Athena-QuickSight Design
Pros
• Athena can search a large dataset stored in S3
• QuickSight has an easy to learn interface
Cons
• Exporting the data to CSV is the only way to share QuickSight reports with
non IAM users
• QuickSight is weak in terms of analytic capability and variety of
visualizations compared to Kibana
• Athena can be slow when searching a large set of data
Security Considerations
• Active sharing of visualizations to other IAM users in QuickSight is
necessary for others to view visualizations

8. ©Practical Code, LLC 2019
Elasticsearch-Kibana Design
Pros
• Kibana has many Business Intelligence options already created from which
to choose
• Elasticsearch’s ability to run queries is fast and the data is redundant for
multiple read
• With Kibana, users can search the data for specific events or strings
Cons
• Learning curve to use Kibana’s interface
• Elasticsearch is an on-going charge not pay per usage
Security Considerations
• Cognito enables users to log in to Kibana with a username and password

9. ©Practical Code, LLC 2019
Operations Costs

10. ©Practical Code, LLC 2019
Athena-QuickSight Design Operating Costs
Operating Costs per
Month
Charges Low Traffic High Traffic
S3 Storage Charges:
$0.023 per GB
At 60MB:
(2($0.00138)) = $0.00276
At 600MB:
(2($0.0138)) = $0.0276
Lambda Compute Charges: $0.00001667
per GB-sec.
Request Charges:
$0.20 per 1 million requests.
Memory: 128 MB
Executed: 120,000 times
Ran: 200ms each time
Compute Charges: $0.05
Request Charges: $0.024
Total: $0.074
Memory: 128 MB
Executed: 1.2 million times
Ran: 200ms each time
Compute charges: $0.50
Request charges: $0.24
Total: $0.74
Athena TB of data scanned per query:
$5.00.
1 TB:
1TB * $5.00 = $5.00
1 TB:
1TB * $5.00 = $5.00
QuickSight Charge per user: $12.00 1 User: $12.00 1 User: $12.00
Total per Month $17.08 $17.77

11. ©Practical Code, LLC 2019
Elasticsearch-Kibana Design Operating Costs
Operating Costs per
Month
Charges Low Traffic High Traffic
S3 Storage Charges: $0.023 GB At 60MB: $0.00138 At 600MB: $0.0138
Lambda Compute Charges:
$0.00001667 per GB-sec
Request Charges:
$0.0000002 per request
Memory: 128 MB
Executed: 120,000 times
Ran: 200ms each time
Compute Charges: $0.05
Request Charges: $0.024
Total: $0.074
Memory: 128 MB
Executed: 1.2 million times
Ran: 200ms each time
Compute charges: $0.50
Request charges: $0.24
Total: $0.74
Elasticsearch and
Kibana
Charges for t2.small.elasticsearch:
$0.036 per hour
750 hours: $27.00 730 hours: $27.00
Cognito Charges per active monthly users:
$0.00550
10 users: $0.055 10 users: $0.055
Total per Month $27.13 $27.81

12. ©Practical Code, LLC 2019
Comparison of Kibana and QuickSight

13. ©Practical Code, LLC 2019
QuickSight vs Kibana Comparison

14. ©Practical Code, LLC 2019
QuickSight Web Logs Visualization

15. ©Practical Code, LLC 2019
Kibana Web Logs Visualization

16. ©Practical Code, LLC 2019
Thank you!
Syenna Graham
syenna.graham@practicalcode.us
540-860-0772