Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports


Published on

This presentation talks about the Following -

-Working of AWS S3 & CloudFront Logs with respect to
Content Storing and Distribution.
-The hidden potential of your Stored S3 & CloudFront Logs
& Unlocking them with Cloudlytics
-Some of our Reports using Cloudlytics

Check the video embedded after the slideshare for a Live recording of our webinar conducted around this topic.

Published in: Technology
  • Be the first to comment

Cloudlytics: In Depth S3 & CloudFront Log Analysis - Featuring Reports

  1. 1. Featuring Our Latest Reports
  2. 2. AGENDA Introduction To Amazon S3 & CloudFront Log Processing Using AWS Vs Traditional ways Log Processing With Cloudlytics- Big Data Approach Cloudlytics Use cases Cloudlytics Reports & Live Demo 2
  3. 3. What is Amazon Simple Storage Service or S3? • Amazon Simple Storage Service is storage for the internet • Stores 2 trillion+ objects • 1.1 million requests per second at peak • Each time a request is made to access a file on S3, an entry to log file is created • Average size of each log entry 550B Image Courtesy: 3
  4. 4. What is Amazon CloudFront? Amazon CloudFront is a web service for content delivery 1 CloudFront decreases latencies 2 for object downloads and streams 5 Average size of each log entry 650B 4 3 Supports static and dynamic content, including web pages 4 Each time a request is made to access a file on CloudFront, an entry to log file is created
  5. 5. Information Hidden in S3 & CloudFront Logs Object Details Download Status Download /Streaming Time Number of Bytes Transferred Details about Edge Locations IP Address of the Requester Referrer Link Time Taken to Download Object(S3) Details about Play, Pause, Stop (Streaming Content on CloudFront) 5 -
  6. 6. Uncover the Hidden Information • Generating logs for Amazon S3 & CloudFront logs is optional S3 logs • Log files are stored in S3 buckets CloudFront logs • CloudFront Log files are compressed Logs Stored in S3 and stored in .gz format • A log file is generated every hour, but we have seen varied patterns with multiple files generated every hour • No ready solution from AWS to process these log files 6 Logs Analyzed by CLOUDLYTICS
  7. 7. Traditional Log Processing • Extract data from the source using an ETL tool • Transform data and load in a data-warehouse • Takes days to process a few GBs of log files using traditional hardware • Alternately use a Hadoop distribution to process logs • But maintaining a Hadoop cluster is a huge overhead 7 Image Courtesy:
  8. 8. Log Processing with Cloudlytics • Cloudlytics - Analyze your Amazon S3 & CloudFront Logs • Detailed analysis of your S3 & CloudFront access patterns • Dynamic Graphs to get a 360 degree perspective • Scalable & Reliable service built using Amazon EMR & RedShift • Pay as you go 8
  9. 9. Log Processing – Big Data Approach • Cloudlytics extracts log files stored in S3 buckets • Processes the log files to transform information • Stores the processed data in a data-warehouse • Graphical and tabular reports generated from data-warehouse 9
  10. 10. Cloudlytics Use Cases 10
  11. 11. Independent Software Vendors (ISVs) • ISVs distribute downloadable software to end users across the globe • ISVs need to ensure that downloads are fast, helps improve user experience • ISVs need to track each download for success and failure • Identify broken links on the website, helps improve user experience • Identify the most popular downloads, focus on popular products • Identify spam attacks, help reduce bandwidth costs 11
  12. 12. E-Learning Companies • E-Learning companies distribute educational content in multiple formats (ebooks, audio, video etc.) • Figure out the most popular content • Figure out end user engagement by querying number of events per request (Play, Pause, Stop) Image Courtesy: • Get a breakup of requests by Operating System and Devices, develop content creation for specific platforms 12
  13. 13. Media Organizations • Large number of media assets available online • Content does not go any changes during it’s life cycle • Some content is extremely popular while others do not get any views • Identify the most popular content, set caching mechanism • Figure out end user engagement by querying number of events per request (Play, Pause, Stop) • Identify the edge locations from which the content is downloaded the most, optimize billing using CloudFront pricing Class 13
  14. 14. Cloudlytics REPORTS Which Reports Resonate with Your Business Needs? 14
  15. 15. Geographic Reports 15
  16. 16. Browser & OS Statistics 16
  17. 17. Detail IP Monitoring 17
  18. 18. Timeline Charts 18
  19. 19. The TOP 10 19
  20. 20. Edge Location Traffic 20
  21. 21. DEMO Let’s Look at Cloudlytics in Action Image Courtesy: 21
  22. 22. How to get started? So Where do you Get Started ?? Image courtesy: http://blogs.position2 .com/best-of-the-week-august-24-2012 22
  23. 23. Get Started in 3 Easy Steps Register for Free Configure your Log buckets Analyze & Generate Reports 23
  24. 24. Pricing • No upfront costs • Register for free and analyze upto 25MB logs/month FREE • Pay only for the amount of logs you subscribe 24
  25. 25. Advantage - Cloudlytics • Scalable & Reliable • Developed using Amazon Web Services tools like Amazon EMR & Amazon Redshift • Developed by BlazeClan Technologies, leading Consulting Partner with Amazon Web Services • Pay as you go service with no contracts and no lock-ins 25
  26. 26. To sum it Up Identifying Popular Downloads & Streams Identify Edge Location Traffic and optimize AWS Billing Get Geographical Distribution of Downloads & Streams Identify Spam Attacks Get Streaming Pattern for Video Content Improve User Experience by Calculating & Optimizing Latencies 26
  27. 27. Our Global User Reach 80+ Users and Counting! 27
  28. 28. Coming Soon! Image Courtesy: 28
  29. 29. Upcoming Webinar Check out Our Upcoming Webinars @ 29
  30. 30. Thank you Visit Contact: Follow Us On : Our Blog :