Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Analyze Amazon CloudFront and Lambda@Edge Logs to Improve Customer Experience (CTD410) - AWS re:Invent 2018

479 views

Published on

Nowadays, web servers are often fronted by a global content delivery network, such as Amazon CloudFront, to accelerate delivery of websites, APIs, media content, and other web assets. In this hands-on-workshop, learn to improve website availability, optimize content based on devices, browser and user demographics, identify and analyze CDN usage patterns, and perform end-to-end debugging by correlating logs from various points in a request-response pipeline. Build an end-to-end serverless solution to analyze Amazon CloudFront logs using AWS Glue and Amazon Athena, generate visualization to derive deeper insights using Amazon QuickSight, and correlate with other logs such as CloudWatch logs to provide finer debugging experiences. Discuss how you can extend the pipeline you just built to generate deeper insights needed to improve the overall experience for your users.

  • Be the first to comment

Analyze Amazon CloudFront and Lambda@Edge Logs to Improve Customer Experience (CTD410) - AWS re:Invent 2018

  1. 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Welcome to re:Invent 2018! CTD410 • Please collect the credit coupon. You can apply this coupon towards completing the labs in this workshop. • Please do not use your production account for the labs. • Choose EU-West-1(Ireland) region for the labs • Navigate to the following web link for workshop lab instruction https://github.com/aws-samples/amazon-cloudfront-log-analysis • Deploy the solution using CloudFormation template by completing the following 2 steps • Create a Key Pair for EC2 Instances • Deploy Solution
  2. 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Analyze Amazon CloudFront & Lambda@Edge Logs to Improve Customer Experience Gowri Balasubramanian Sr. Solution Architect Amazon Web Services C T D 4 1 0 Rajeev Srinivasan Strategic Solution Architect Amazon Web Services
  3. 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda Improving customer experience Amazon CloudFront Lambda@Edge Customer reference Log collection Lab 1: Serverless analysis pipeline • Building block overview—AWS Glue, Amazon Athena, Amazon QuickSight • Pipeline architecture Lab 2: Geospatial visualization pipeline • Building block overview—Logstash, Amazon Elasticsearch, Kibana • Pipeline architecture
  4. 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Improving customer experience Increase service and application availability Meet compliance & audit requirements Reduce website latency Content optimization Bot detection & mitigation
  5. 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  6. 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon CloudFront – Access logs AWS Cloud Region VPC Application servers On-Premise Database servers Application servers Webserver servers Routed to the nearest PoP Dynamic content Edge Location Global users
  7. 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lambda@Edge Edge location Viewer request Viewer response Origin request Origin response Origin server Devices
  8. 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lambda@Edge – Authoring Author @ Lambda console Lambda function Region = us-east-1 Create Lambda function Region Lambda function Region Lambda function Region Replicate Trigger association
  9. 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lambda@Edge – Improved user experience Real-time image transformation A/B testing User AuthN & AuthZ User prioritization
  10. 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lambda@Edge – User tracking & analytics US EU @Edge Request Response Triggered
  11. 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  12. 12. Social Media, Digital Advertising, EdTech, Finance Others E-commerce Media & Entertainment Gaming CloudFront customers
  13. 13. “By combining the power of AWS compute and storage with the Amazon CloudFront content delivery network, we can provide an excellent user experience to everybody playing our games and watching our video content—regardless of their geographical location.” Mikko Peltola Operations Lead
  14. 14. “Thanks to Lambda@Edge and CloudFront, DataDome simplified the onboarding process for our customers, who can activate our bot protection in one click in their AWS console. We made our service available globally by deploying our solution across AWS edge locations. Integrating Lambda@Edge and CloudFront also allowed our company—based in France—to gain a global reach.” Benjamin Fabre DataDome co-founder & CTO
  15. 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  16. 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Edge location Log collection from the life of a user request VPC Application servers AWS Cloud Client devices Viewer request Viewer response Origin request Origin response Load balancer logs Amazon S3 access logs Applicationlogs Lambda@Edge logs Lambda@Edge logs Clientsidelogs CloudFront access logs
  17. 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Edge location Log collection VPC Application servers AWS Cloud Client devices Viewer request Viewer response Origin request Origin response Load balancer logs Lambda@Edge logs CloudFront access logs
  18. 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  19. 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Service overview – AWS Glue JDBC Data source JDBC Data target AWS Glue Crawler AWS Glue Data Catalog AWS Glue ETL Scan data read data readmetadata writemetadata write transformed data How AWS Glue works • Build your Data Catalog • Generate & edit transformations • Schedule and run your jobs AWS Glue components AWS Glue Crawler AWS Glue Data Catalog AWS Glue ETL
  20. 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Service overview – Amazon Athena • Amazon Athena is an interactive query service that makes it easy to analyze data directly from Amazon Simple Storage Service (Amazon S3) using standard SQL Data Catalog visualize
  21. 21. Service overview – Amazon QuickSight Amazon QuickSight is natively integrated with AWS data sources, as well as on-premise and hosted databases and third-party business applications On-premises Securely connect to on-premise databases and flat files like Excel and CSV In the cloud Connect to hosted database, big data formats, and secure VPCs Applications Connect directly to third- party business applications • Salesforce • Square • Adobe Analytics • Jira • ServiceNow • Twitter • GitHub • Amazon Redshift • Amazon Relational Database Service (Amazon RDS) • Amazon S3 • Athena • Amazon Aurora • Teradata • MySQL • Presto • Spark • SQL Server • PostgreSQL • MariaDB • Snowflake • Excel • CSV • Teradata • MySQL • SQL Server • PostgreSQL © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  22. 22. Serverless analysis pipeline – Architecture Raw CloudFront access logs Lambda@Edge logs ALB logs Optimized CloudFront access logs Optimized Lambda@Edge logs Optimized ALB logs your Amazon S3 bucket containing processed logs © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  23. 23. Serverless analysis pipeline—Workshop Optimized CloudFront access logs Optimized Lambda@Edge logs Optimized ALB logs your Amazon S3 Bucket containing processed logs © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  24. 24. Serverless analysis pipeline — Workshop © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. time [timestamp] location [string bytes [bigint] requestip [string] method [string] host [string] uri [string] status [int] referrer [string] useragent [string] querystring [string] requestid [string] cookie [string] resulttype [string] hostheader [string] requestprotocol [string] requestbytes [bigint] timetaken [double] xforwardedfor [string] sslprotocol [string] sslcipher [string] responseresulttype [string] httpversion [string] year [string] month [string] day [string] requestid [string] executionregion [string] distributionid [string] distributionname [string] eventtype [string] requestdata [string] customtraceid [string] useragentstring [string] year [string] month [string] day [string] hour [string] requestid [string] executionregion [string] distributionid [string] distributionname [string] eventtype [string] requestdata [string] customtraceid [string] useragentstring [string] deviceformfactor [string] viewercountry[string] year [string] month [string] day [string] hour [string] trace_id [string] type [string] time [timestamp] elb [string] client_ip_port [string] target_ip_port [string] request_processing_time [double] target_processing_time [double] response_processing_time [double] elb_status_code [string] target_status_code [string] received_bytes [bigint] sent_bytes [bigint] request_verb [string] request_url [string] request_proto [string] user_agent [string] ssl_cipher [string] ssl_protocol [string] target_group_arn [string] domain_name [string] chosen_cert_arn [string] region [string] year [string] month[string] day[string] requestid [string] trace_id [string] received_bytes [int] distributionname [string] executionregion [string] distributionid [string] location [string] sent_bytes [int] responseresulttype [string] xforwardedfor [string] type [string] customtraceid [string] querystring [string] client_ip_port [string] response_processing_time [double] elb [string] deviceformfactor [string] elb_status_code [string] uri [string] request_verb [string] col24 [string] request_url [string] region [string] hostheader [string] request_processing_time [double] resulttype [string] method [string] useragent [string] httpversion [string] target_status_code [string] target_ip_port [string] requestdata [string] host [string] referrer [string] cookie [string] bytes [bigint] target_processing_time [double] alb_time [timestamp] viewercountry [string] timetaken [double] requestbytes [bigint] target_group_arn [string] sslprotocol [string] requestprotocol [string] status [int] time [timestamp] requestip [string] sslcipher [string] request_proto [string] col25 [string] user_agent [string] year [string] month [string] day [string]
  25. 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  26. 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Elasticsearch Service Fully managed service that is easy to deploy, manage, and scale + Fully managed
  27. 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Benefits of Amazon Elasticsearch Service Supports open-source APIs and tools Drop-in replacement with no need to learn new APIs or skills Easy to use Deploy a production-ready Amazon ES cluster in minutes Scalable Resize your cluster with a few clicks or a single API call Secure Deploy into your VPC and restrict access using security groups and AWS Identity and Access Management (IAM) policies Highly available Replicate across Availability Zones, with monitoring and automated self-healing Tightly integrated with other AWS services Seamless data ingestion, security, auditing, and orchestration
  28. 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Logstash simplifies transformation of CloudFront logs Logstash is open-source pipeline that can be installed on Amazon Elastic Compute Cloud (Amazon EC2) instance Logstash uses the bulk API to upload data into Amazon ES domain using Amazon S3 plugin (input) and logstash-output-amazon-es plugin (output) Grok pattern matching format/transform CloudFront log fields, for example: • Combine date and time • Extract user agent information GeoIP plugin provides geo-coordinates based on the requestip Logstash
  29. 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Geospatial visualization pipeline – Architecture Logstash Amazon ES data nodes Kibana Access logs Buffer Transform Deliver Amazon ES master nodes Analytics UI Poll logs
  30. 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Cloud AWS Region Public subnet1 Availability zone 1 Public subnet2 Availability zone 2 VPC Kibana Internet gateway LogstashProxy Server Lab2 Deployment Architecture
  31. 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Deriving insights from ELK-based pipeline Customer experience • Geo-spatial visualization • Global distribution of your requests • Insights on which location users experience increased latency Debugging use cases • Are customers receiving errors from specific devices? • Do certain objects have more errors than others? • Total number of hits vs. misses vs. errors for a given time period
  32. 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Related breakouts Thursday, Nov 29 Customizing Content Delivery with Labmda@Edge – CTD415-R1 2:30 P.M. – 5:00 P.M. | Bellagio, Level 1, Grand Ballroom 6 Thursday, Nov 29 Use Monitoring, Logs, and Analytics Tools to Measure CDN And Site Performance – CTD207-R3 11:30 P.M. – 12:30 P.M. | Mirage, Grand Ballroom D
  33. 33. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Gowri Balasubramanian Rajeev Srinivasan
  34. 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  35. 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Workshop CTD410 • Please collect the credit coupon. You can apply this coupon towards completing the labs in this workshop. • Please do not use your production account for the labs. • Choose EU-West-1(Ireland) region for the labs • Navigate to the following web link for workshop lab instruction https://github.com/aws-samples/amazon-cloudfront-log-analysis • Deploy the solution using CloudFormation template by completing the following 2 steps • Create a Key Pair for EC2 Instances • Deploy Solution

×