Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Amazon CloudWatch Logs and AWS Lambda: A Match Made in Heaven

In this session, we cover three common scenarios that include Amazon CloudWatch Logs and AWS Lambda. Learn how to build an Elasticsearch cluster from historical data using Amazon S3, Lambda, and CloudWatch Logs. Discover how to add details to CloudWatch alarm notifications using Amazon SNS and Lambda. Finally, understand how to bring Elastic Load Balancing logs to CloudWatch Logs using S3 bucket triggers from Lambda.

Speaker: Leo Zhadanovsky, Principle Solutions Architect. Amazon Web Services
Level: 300

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

Amazon CloudWatch Logs and AWS Lambda: A Match Made in Heaven

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon CloudWatch Logs and AWS Lambda: A Match Made in Heaven Leo Zhadanovsky Principal Solutions Architect, AWS September 12, 2017
  2. 2. What to Expect from the Session Look at industry trends impacting monitoring Learn about Amazon CloudWatch and Amazon CloudWatch Logs Understand several key monitoring use cases See Amazon CloudWatch and AWS Lambda in action
  3. 3. Centralize Centralize logs from Elastic Load Balancing (ELB) using S3 bucket triggers Customize Customize alarms from Amazon CloudWatch to fit your specific needs Analyze Build an on-demand, scalable Amazon Elasticsearch Service cluster to solve a specific problem or to do analysis What to expect: Scenario preview
  4. 4. Recognize this?
  5. 5. Day in the life! The story you are about to hear is true (mostly)…Only the names have been changed to protect the innocent…
  6. 6. A customer writes a high severity ticket – your Application, ImportantApp, is down. John, the on-call developer, is paged through the ticketing system. None of your alarms fired. Blissful ignorance
  7. 7. John engages and starts to scan service dashboards. He does see intermittent availability impact, but doesn’t know how to assess impact to customers or where to begin troubleshooting. He decides to escalate to a manager on call. Confusion
  8. 8. More customer tickets are pouring in. An escalation manager, Jane, joins the event and starts to assess the situation and impact. John and Jane’s CTO happens to notice the problem. Sends Jane an IM – “Jane, what’s going on with ImportantApp?” Stress
  9. 9. Jane and John recall a recent issue where certain customers started to issue “expensive” operations. John starts log diving on their production hosts. John identifies a suspect customer. Jane cuts a ticket and John prepares a configuration change to block the customer. False hope
  10. 10. The other team engages and indicates they didn’t change anything. Jane and John also confirm this when the availability impact persists after deploying the configuration change. Out of ideas, John suggests to fail over to the standby – “It can’t hurt…” Desperation
  11. 11. After the failover, ImportantApp recovers. (Yay!) Our root cause deep dive finds that a new JDBC version introduced a memory leak leading to Java heap exhaustion. We fix the leak, add new alarms on memory usage, and tune our service alarms. Enlightenment
  12. 12. Can we do better than that?
  13. 13. Day in the life - Reflection • We have missing alarms, and some of the alarms we have are not actionable. • We don’t always have the right logs and interacting with them can be tedious. • Our dashboards do not tell us enough about customer impact or behavior changes. Monitoring is really (really) hard!
  14. 14. Trends in monitoring
  15. 15. Trend: Complexity increasing • Distributed microservices based applications • Applications are written in different languages and frameworks • Workloads are increasingly running on transient resources such as containers and serverless compute • Specialization in persistence tier
  16. 16. • Small changes are continuously built, tested, and deployed • As the scale and design of applications are changing rapidly, so are the infrastructure needs • Applications are global and customer behavior is unpredictable • Increased role of automation Trend: Applications are more dynamic
  17. 17. • Increased role of applications in business outcomes (revenue, cost, SLA) • Rapidly evolving applications are required to gain competitive advantage • Increased expectations from customers Trend: More business impact
  18. 18. • Monitoring is no longer standalone. It is one aspect of management and increasingly integrated into the lifecycle of the application Trend: Integrated management
  19. 19. Amazon CloudWatch See React Diagnose Resolve
  20. 20. Use AWS generated metrics, logs, and events over time to understand the behavior of your system Publish custom metrics, logs, and events for your application specific telemetry See React Diagnose Resolve
  21. 21. Trigger automatic notifications based on your own rules and metric thresholds See React Diagnose Resolve
  22. 22. Inspect, navigate, zoom, and correlate across time to investigate issues Jump to your logs directly from your metrics to perform searches or generate additional metrics from log data See React Diagnose Resolve
  23. 23. Easily and automatically correct issues using common actions that you control Define your own custom actions based on Lambda functions for more fine-grained control See React Diagnose Resolve
  24. 24. • Metrics price drop • More metrics, logs, events from AWS services: AWS CloudTrail, AWS Elastic Beanstalk, Amazon SES • Simple navigation from metrics to your logs • Upgraded metric retention from 2 weeks to up to 15 months Recent improvements
  25. 25. • Support for arbitrary metric percentiles • collectd output plugin to simplify metric collection • Improvements in dashboards (new widgets, dark theme, Y axis limits) • Improved Logs console experience • 1 Second Metrics Resolution • How to treat missing data Recent improvements (continued)
  26. 26. Not just about what’s inside CloudWatch • Monitoring is hard (very hard) • Every enterprise, team and situation has unique needs • We have a rich partner community • We give you the tools and flexibility to integrate with other AWS services
  27. 27. Flexible Egress Comprehensive Ingress Amazon EC2 AWS CloudTrail Amazon ECS AWS Lambda Amazon ES Amazon S3 AWS Lambda AMAZON CLOUDWATCH LOGS Centralize Search Monitor Stream Retain Secure A log service for arbitrary scale, availability, durability, and security AWS Kinesis Custom
  28. 28. Serverless compute: AWS Lambda COMPUTE SERVICE EVENT DRIVEN Run arbitrary code without managing servers Code only runs when it needs to run
  29. 29. Centralize Centralize logs from Elastic Load Balancing (ELB) using Amazon S3 bucket triggers Customize Customize alarms from Amazon CloudWatch to fit your specific needs Analyze Build an on-demand, scalable Amazon Elasticsearch Service cluster to solve a specific problem or to do analysis
  30. 30. Problem statements • Log data is scattered on instances and Amazon S3 buckets • It would be better if it were centralized in CloudWatch Logs for searching and filtering • Today CloudWatch provides an agent for instance logs; what about Amazon S3 delivered logs?
  31. 31. Flow of events LambdaS3ELB CloudWatch Logs ELB logs archived in Amazon S3 S3 Object Create Event Notification to AWS Lambda AWS Lambda reads the ELB log from Amazon S3 and publishes to CloudWatch Logs
  32. 32. Apache Servers ELB & Lambda Configured Lambda Triggered ELB logs in CloudWatch Provision three Amazon EC2 instances running Apache
  33. 33. Apache Servers ELB & Lambda Configured Lambda Triggered ELB logs in CloudWatch Create a load balancer
  34. 34. Apache Servers ELB & Lambda Configured Lambda Triggered ELB logs in CloudWatch Create a target group with the three Apache servers
  35. 35. Apache Servers ELB & Lambda Configured Lambda Triggered ELB logs in CloudWatch Enable ELB log delivery to Amazon S3
  36. 36. Apache Servers ELB & Lambda Configured Lambda Triggered ELB logs in CloudWatch Configure the Lambda function to trigger on S3 object create
  37. 37. Apache Servers ELB & Lambda Configured Lambda Triggered ELB logs in CloudWatch Read the ELB logs from the S3 bucket by invoking S3 GetObject API
  38. 38. Apache Servers ELB & Lambda Configured Lambda Triggered ELB logs in CloudWatch Post the logs into CloudWatch Logs by invoking putLogEvents SDK API
  39. 39. Apache Servers ELB & Lambda Configured Lambda Triggered ELB logs in CloudWatch ELB serving traffic
  40. 40. Apache Servers ELB & Lambda Configured Lambda Triggered ELB logs in CloudWatch Verify that ELB logs are delivered to S3
  41. 41. Apache Servers ELB & Lambda Configured Lambda Triggered ELB logs in CloudWatch Access the ELB logs from CloudWatch Logs
  42. 42. Apache Servers ELB & Lambda Configured Lambda Triggered ELB logs in CloudWatch Search for the HTTP GET calls in the last 24 hours
  43. 43. Apache Servers ELB& Lambda Configured Lambda Triggered ELB logs in CloudWatch Create a filter pattern to extract requests with a latency of more than 1 ms
  44. 44. Define a metric filter on the log group Apache Servers ELB& Lambda Configured Lambda Triggered ELB logs in CloudWatch
  45. 45. Apache Servers ELB& Lambda Configured Lambda Triggered ELB logs in CloudWatch View the metric graph for ELB latency HighLatencyCount
  46. 46. Key takeaways 1. Amazon S3 delivered log data from any source can be centralized into CloudWatch Logs using Lambda 2. You can search and extract metrics from those logs in near real time
  47. 47. Centralize Centralize logs from Elastic Load Balancing (ELB) using S3 bucket triggers Customize Customize alarms from Amazon CloudWatch to fit your specific needs Analyze Build an on-demand, scalable Amazon Elasticsearch Service cluster to solve a specific problem or to do analysis
  48. 48. Problem Statements • When you get an alarm, you want enough information to decide whether it needs immediate attention or not • You want to customize the alarm text and format to your operational needs
  49. 49. Flow of Events Error in Logs generates an alarm to an SNS topic Amazon SNS topic triggers an Event Notification to Lambda AWS Lambda gets the filtered logs, calls SES to email with those logs Amazon SESAWS LambdaCloudWatch Logs CloudWatch Alarms
  50. 50. Metric Filter Defined Alarm & Lambda Configured Lambda Triggered SES sends Email Provision an EC2 instance with Tomcat running on it Hello World
  51. 51. CloudWatch agent sends EC2 instance logs to CloudWatch Logs Metric Filter Defined Alarm & Lambda Configured Lambda Triggered SES sends Email
  52. 52. Define a filter pattern to extract unauthorized access attempts Metric Filter Defined Alarm & Lambda Configured Lambda Triggered SES sends Email
  53. 53. Define a metric filter on the log group Metric Filter Defined Alarm & Lambda Configured Lambda Triggered SES sends Email
  54. 54. Define an alarm with a specific threshold for that metric Metric Filter Defined Alarm & Lambda Configured Lambda Triggered SES sends Email test
  55. 55. Configure the Lambda function to trigger on SNS topic message Metric Filter Defined Alarm & Lambda Configured Lambda Triggered SES sends Email Hello World
  56. 56. Metric Filter Defined Alarm & Lambda Configured Lambda Triggered SES sends Email Get the metric filter information by invoking the describeMetricFilters SDK API
  57. 57. Metric Filter Defined Alarm & Lambda Configured Lambda Triggered SES sends Email Get the relevant log data by invoking the filterLogEvents SDK API
  58. 58. Metric Filter Defined Alarm & Lambda Configured Lambda Triggered SES sends Email SES sends an email with the relevant log data
  59. 59. Unauthorized access attempts on the Tomcat server Metric Filter Defined Alarm & Lambda Configured Lambda Triggered SES sends Email
  60. 60. Alarm generated due to unauthorized access attempts Metric Filter Defined Alarm & Lambda Configured Lambda Triggered SES sends Email test test test test
  61. 61. View email with the relevant log data about the alarm Metric Filter Defined Alarm & Lambda Configured Lambda Triggered SES sends Email test test Hell o test
  62. 62. Key takeaways • Alarms can be customized to add specific details about the issue • When you see a spike on a metric, you can also get the logs describing the issue triggering the alarm • The Lambda function can be extended to add your specific information to the alarm
  63. 63. Centralize Centralize logs from Elastic Load Balancing (ELB) using S3 bucket triggers Customize Customize alarms from Amazon CloudWatch to fit your specific needs Analyze Build an on-demand, scalable Amazon Elasticsearch Service cluster to solve a specific problem or to do analysis
  64. 64. Problem Statements • You want to do log analysis using Amazon Elasticsearch Service but don’t want to leave the cluster running all the time • You want to send data to Amazon Elasticsearch Service, but don’t want to manage ongoing operations • Build an on-demand Elasticsearch cluster from historical data
  65. 65. Flow of events LambdaS3 CloudWatch Logs CloudWatch Logs in a specific timeframe exported to S3 S3 Object Create Event Notification to Lambda AWS Lambda transforms and sends only filtered logs to Amazon Elasticsearch Service Amazon ES
  66. 66. Logs exported to S3 Lambda Configured Logs to Elasticsearch Visuals in Kibana Configure flow logging for your VPC
  67. 67. Logs exported to S3 Lambda Configured Logs to Elasticsearch Visuals in Kibana View the VPC Flow Logs in CloudWatch Logs
  68. 68. Logs exported to S3 Lambda Configured Logs to Elasticsearch Visuals in Kibana Create an Amazon ES domain
  69. 69. Logs exported to S3 Lambda Configured Logs to Elasticsearch Visuals in Kibana Read the VPC Flow Logs from the S3 bucket by invoking GetObject API
  70. 70. Logs exported to S3 Lambda Configured Logs to Elasticsearch Visuals in Kibana Transform the VPC Flow Logs into a JSON document for Amazon ES
  71. 71. Logs exported to S3 Lambda Configured Logs to Elasticsearch Visuals in Kibana Ingest the logs into Amazon ES by putting to its HTTP endpoint
  72. 72. Logs exported to S3 Lambda Configured Logs to Elasticsearch Visuals in Kibana Export the VPC Flow Logs to Amazon S3
  73. 73. Logs exported to S3 Lambda Configured Logs to Elasticsearch Visuals in Kibana VPC Flow Logs exported to Amazon S3
  74. 74. Logs exported to S3 Lambda Configured Logs to Elasticsearch Visuals in Kibana View the VPC Flow Logs in Kibana
  75. 75. Logs exported to S3 Lambda Configured Logs to Elasticsearch Visuals in Kibana View different dashboards in Kibana
  76. 76. Logs exported to S3 Lambda Configured Logs to Elasticsearch Visuals in Kibana View different dashboards in Kibana
  77. 77. Key takeaways • Send historical data within a timeframe to Amazon Elasticsearch Service on demand • This reduces cost, burden of scalability, and operations time • Troubleshooting gets easier because you have only limited and relevant data
  78. 78. Recap • Monitoring is more important than ever, but still too hard • Amazon CloudWatch is working to make monitoring easier • Amazon CloudWatch Logs and AWS Lambda are powerful tools for tailoring your monitoring for your business needs
  79. 79. Useful Links • CloudWatch Overview - https://aws.amazon.com/cloudwatch/ • Documentation - https://aws.amazon.com/documentation/cloudwatch/ • CloudWatch Blog - https://aws.amazon.com/blogs/aws/category/amazon- cloud-watch/ • Lambda functions used in the demo scenarios Centralize - https://github.com/awslabs/cloudwatch-logs-centralize-logs Customize - https://github.com/awslabs/cloudwatch-logs-customize-alarms Analyze - https://github.com/awslabs/cloudwatch-logs-analyze-data
  80. 80. Remember to complete your evaluations!
  81. 81. Thank you!

×