Nowadays, web servers are often fronted by a global content delivery network, such as Amazon CloudFront, to accelerate delivery of websites, APIs, media content, and other web assets. In this hands-on-workshop, learn to improve website availability, optimize content based on devices, browser and user demographics, identify and analyze CDN usage patterns, and perform end-to-end debugging by correlating logs from various points in a request-response pipeline. Build an end-to-end serverless solution to analyze Amazon CloudFront logs using AWS Glue and Amazon Athena, generate visualization to derive deeper insights using Amazon QuickSight, and correlate with other logs such as CloudWatch logs to provide finer debugging experiences. Discuss how you can extend the pipeline you just built to generate deeper insights needed to improve the overall experience for your users.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Welcome to re:Invent 2018! CTD410
• Please collect the credit coupon. You can apply this coupon towards completing the labs in this
workshop.
• Please do not use your production account for the labs.
• Choose EU-West-1(Ireland) region for the labs
• Navigate to the following web link for workshop lab instruction
https://github.com/aws-samples/amazon-cloudfront-log-analysis
• Deploy the solution using CloudFormation template by completing the following 2 steps
• Create a Key Pair for EC2 Instances
• Deploy Solution

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Analyze Amazon CloudFront & Lambda@Edge
Logs to Improve Customer Experience
Gowri Balasubramanian
Sr. Solution Architect
Amazon Web Services
C T D 4 1 0
Rajeev Srinivasan
Strategic Solution Architect
Amazon Web Services

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
Improving customer experience
Amazon CloudFront
Lambda@Edge
Customer reference
Log collection
Lab 1: Serverless analysis pipeline
• Building block overview—AWS Glue, Amazon Athena, Amazon QuickSight
• Pipeline architecture
Lab 2: Geospatial visualization pipeline
• Building block overview—Logstash, Amazon Elasticsearch, Kibana
• Pipeline architecture

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Improving customer experience
Increase service and application
availability
Meet compliance & audit
requirements
Reduce website latency Content optimization Bot detection & mitigation

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon CloudFront – Access logs
AWS Cloud Region
VPC
Application
servers
On-Premise
Database
servers
Application
servers
Webserver
servers
Routed to the nearest PoP
Dynamic content
Edge Location
Global users

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lambda@Edge
Edge location
Viewer request
Viewer response
Origin request
Origin response
Origin server
Devices

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lambda@Edge – Authoring
Author @
Lambda console
Lambda function
Region = us-east-1
Create
Lambda function
Region
Lambda function
Region
Lambda function
Region
Replicate
Trigger
association

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lambda@Edge – Improved user experience
Real-time image transformation
A/B testing
User AuthN & AuthZ
User prioritization

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lambda@Edge – User tracking & analytics
US
EU
@Edge
Request
Response
Triggered

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

13. Social Media, Digital
Advertising, EdTech,
Finance
Others
E-commerce
Media &
Entertainment
Gaming
CloudFront customers

14. “By combining the power of AWS compute and storage with the Amazon
CloudFront content delivery network, we can provide an excellent user
experience to everybody playing our games and watching our video
content—regardless of their geographical location.”
Mikko Peltola
Operations Lead

15. “Thanks to Lambda@Edge and CloudFront, DataDome simplified the onboarding
process for our customers, who can activate our bot protection in one click in their
AWS console. We made our service available globally by deploying our solution
across AWS edge locations. Integrating Lambda@Edge and CloudFront also allowed
our company—based in France—to gain a global reach.”
Benjamin Fabre
DataDome co-founder & CTO

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Edge location
Log collection from the life of a user request
VPC
Application
servers
AWS Cloud
Client devices Viewer request
Viewer response
Origin request
Origin response
Load balancer logs
Amazon S3 access logs
Applicationlogs
Lambda@Edge logs
Lambda@Edge logs
Clientsidelogs
CloudFront access logs

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Edge location
Log collection
VPC
Application
servers
AWS Cloud
Client devices Viewer request
Viewer response
Origin request
Origin response
Load balancer logs
Lambda@Edge logs
CloudFront access logs

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service overview – AWS Glue
JDBC
Data source
JDBC
Data target
AWS Glue
Crawler
AWS Glue Data
Catalog
AWS Glue ETL
Scan data
read data
readmetadata
writemetadata
write transformed
data
How AWS Glue works
• Build your Data Catalog
• Generate & edit transformations
• Schedule and run your jobs
AWS Glue components
AWS Glue
Crawler
AWS Glue Data
Catalog
AWS Glue ETL

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Service overview – Amazon Athena
• Amazon Athena is an interactive query service that makes it easy to
analyze data directly from Amazon Simple Storage Service (Amazon
S3) using standard SQL
Data Catalog
visualize

22. Service overview – Amazon QuickSight
Amazon QuickSight is natively integrated with AWS data sources, as well as
on-premise and hosted databases and third-party business applications
On-premises
Securely connect to on-premise
databases and flat files like
Excel and CSV
In the cloud
Connect to hosted database, big
data formats, and secure VPCs
Applications
Connect directly to third-
party business applications
• Salesforce
• Square
• Adobe Analytics
• Jira
• ServiceNow
• Twitter
• GitHub
• Amazon Redshift
• Amazon
Relational
Database Service
(Amazon RDS)
• Amazon S3
• Athena
• Amazon Aurora
• Teradata
• MySQL
• Presto
• Spark
• SQL Server
• PostgreSQL
• MariaDB
• Snowflake
• Excel
• CSV
• Teradata
• MySQL
• SQL Server
• PostgreSQL
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

23. Serverless analysis pipeline – Architecture
Raw
CloudFront
access logs
Lambda@Edge
logs
ALB logs
Optimized
CloudFront
access logs
Optimized
Lambda@Edge
logs
Optimized
ALB logs
your Amazon
S3 bucket
containing
processed logs
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. Serverless analysis pipeline—Workshop
Optimized
CloudFront
access logs
Optimized
Lambda@Edge
logs
Optimized
ALB logs
your Amazon
S3 Bucket
containing
processed logs
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

25. Serverless analysis pipeline — Workshop
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
time [timestamp]
location [string
bytes [bigint]
requestip [string]
method [string]
host [string]
uri [string]
status [int]
referrer [string]
useragent [string]
querystring [string]
requestid [string]
cookie [string]
resulttype [string]
hostheader [string]
requestprotocol [string]
requestbytes [bigint]
timetaken [double]
xforwardedfor [string]
sslprotocol [string]
sslcipher [string]
responseresulttype
[string]
httpversion [string]
year [string]
month [string]
day [string]
requestid [string]
executionregion [string]
distributionid [string]
distributionname [string]
eventtype [string]
requestdata [string]
customtraceid [string]
useragentstring [string]
year [string]
month [string]
day [string]
hour [string]
requestid [string]
executionregion [string]
distributionid [string]
distributionname [string]
eventtype [string]
requestdata [string]
customtraceid [string]
useragentstring [string]
deviceformfactor [string]
viewercountry[string]
year [string]
month [string]
day [string]
hour [string]
trace_id [string]
type [string]
time [timestamp]
elb [string]
client_ip_port [string]
target_ip_port [string]
request_processing_time [double]
target_processing_time [double]
response_processing_time [double]
elb_status_code [string]
target_status_code [string]
received_bytes [bigint]
sent_bytes [bigint]
request_verb [string]
request_url [string]
request_proto [string]
user_agent [string]
ssl_cipher [string]
ssl_protocol [string]
target_group_arn [string]
domain_name [string]
chosen_cert_arn [string]
region [string]
year [string]
month[string]
day[string]
requestid [string]
trace_id [string]
received_bytes [int]
distributionname [string]
executionregion [string]
distributionid [string]
location [string]
sent_bytes [int]
responseresulttype [string]
xforwardedfor [string]
type [string]
customtraceid [string]
querystring [string]
client_ip_port [string]
response_processing_time [double]
elb [string]
deviceformfactor [string]
elb_status_code [string]
uri [string]
request_verb [string]
col24 [string]
request_url [string]
region [string]
hostheader [string]
request_processing_time [double]
resulttype [string]
method [string]
useragent [string]
httpversion [string]
target_status_code [string]
target_ip_port [string]
requestdata [string]
host [string]
referrer [string]
cookie [string]
bytes [bigint]
target_processing_time [double]
alb_time [timestamp]
viewercountry [string]
timetaken [double]
requestbytes [bigint]
target_group_arn [string]
sslprotocol [string]
requestprotocol [string]
status [int]
time [timestamp]
requestip [string]
sslcipher [string]
request_proto [string]
col25 [string]
user_agent [string]
year [string]
month [string]
day [string]

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Elasticsearch Service
Fully managed service that is easy to deploy, manage, and scale
+
Fully managed

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Benefits of Amazon Elasticsearch Service
Supports open-source
APIs and tools
Drop-in replacement with no need
to learn new APIs or skills
Easy to use
Deploy a production-ready
Amazon ES cluster in minutes
Scalable
Resize your cluster with a few
clicks or a single API call
Secure
Deploy into your VPC and
restrict access using security
groups and AWS Identity and
Access Management (IAM) policies
Highly available
Replicate across Availability
Zones, with monitoring and
automated self-healing
Tightly integrated with
other AWS services
Seamless data ingestion, security,
auditing, and orchestration

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Logstash simplifies transformation of CloudFront logs
Logstash is open-source pipeline that can be installed
on Amazon Elastic Compute Cloud (Amazon EC2)
instance
Logstash uses the bulk API to upload data into
Amazon ES domain using Amazon S3 plugin (input)
and logstash-output-amazon-es plugin (output)
Grok pattern matching format/transform CloudFront
log fields, for example:
• Combine date and time
• Extract user agent information
GeoIP plugin provides geo-coordinates based on the
requestip
Logstash

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Geospatial visualization pipeline – Architecture
Logstash
Amazon ES
data nodes
Kibana
Access
logs
Buffer
Transform
Deliver
Amazon ES
master nodes
Analytics UI
Poll
logs

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Cloud
AWS Region
Public subnet1
Availability zone 1
Public subnet2
Availability zone 2
VPC
Kibana
Internet
gateway
LogstashProxy Server
Lab2 Deployment Architecture

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deriving insights from ELK-based pipeline
Customer experience
• Geo-spatial visualization
• Global distribution of your requests
• Insights on which location users experience increased latency
Debugging use cases
• Are customers receiving errors from specific devices?
• Do certain objects have more errors than others?
• Total number of hits vs. misses vs. errors for a given time period

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Related breakouts
Thursday, Nov 29
Customizing Content Delivery with Labmda@Edge – CTD415-R1
2:30 P.M. – 5:00 P.M. | Bellagio, Level 1, Grand Ballroom 6
Thursday, Nov 29
Use Monitoring, Logs, and Analytics Tools to Measure CDN And Site
Performance – CTD207-R3
11:30 P.M. – 12:30 P.M. | Mirage, Grand Ballroom D

34. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Gowri Balasubramanian
Rajeev Srinivasan

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Workshop CTD410
• Please collect the credit coupon. You can apply this coupon towards completing the labs in this workshop.
• Please do not use your production account for the labs.
• Choose EU-West-1(Ireland) region for the labs
• Navigate to the following web link for workshop lab instruction
https://github.com/aws-samples/amazon-cloudfront-log-analysis
• Deploy the solution using CloudFormation template by completing the following 2 steps
• Create a Key Pair for EC2 Instances
• Deploy Solution