The document discusses gaining operational insights from logs by loading streaming log data into Amazon Elasticsearch Service. It describes common log sources on AWS including CloudFront access logs, S3 server access logs, ELB access logs, CloudTrail, and VPC flow logs. It then explains how to load streaming data from these sources into Amazon ES using AWS Lambda from Amazon S3, CloudWatch Logs, Kinesis Firehose, or Logstash. The presentation concludes with a demo of a real world scenario.

1. Gaining Operational Insights
out of your logs
Kobi Biton, Solutions Architect

2. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Common Log sources on AWS
• Loading Streaming data into Amazon
Elasticsearch Service
• Demo: Real world scenario

3. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
We want to turn this…
{
"eventTime": "2016-06-06T09:59:55Z",
"eventSource": "signin.amazonaws.com",
"eventName": "ConsoleLogin",
"awsRegion": "us-east-1",
"sourceIPAddress": ”169.254.169.254",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0)
Gecko/20100101 Firefox/38.0",
"errorMessage": "Failed authentication",
"requestParameters": null,
"responseElements": {
"ConsoleLogin": "Failure"
},
"additionalEventData": {
"LoginTo": "SomeInfo"
"MFAUsed": "Yes"
},
{...}

4. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Into this…

5. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
And then optionally this…
Amazon
SNS
Topic
AWS
Lambda
Cloudwatch
Logs
Alarms

6. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Common Log Sources
• Amazon Cloudfront access logs , S3 server
access logs
• Elastic Load Balancer access logs
• AWS Cloudtrail , VPC Flow Logs
• Amazon SNS notifications
• Amazon EC2 OS Logs / Custom application
logs

7. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Loading Streaming
Data Into Amazon
Elasticsearch Service

8. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
From Amazon S3
Amazon ES
Domain
AWS
Lambda
ELB Logs
Cloudfront
Access Logs
S3 Access
Logs
S3

9. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS
Cloudtrail
VPC Flow
Logs
Cloudwatch
Logs
Amazon ES
Domain
AWS
Lambda
From Amazon Cloudwatch Logs
Cloudwatch Alarms

10. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kinesis Firehose
Agent
Amazon
Kinesis
Firehose
Amazon ES
Domain
Via Amazon Kinesis Firehose

11. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Logstash
Agent
Log Objects
Amazon ES
Domain
Via Logstash Agent

12. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DEMO

14. kobibito@amazon.com

15. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Appendix

16. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Useful Links:
• https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-
aws-integrations.html
• https://github.com/awslabs/cloudwatch-logs-subscription-
consumer/tree/master/configuration/kibana
• https://docs.aws.amazon.com/firehose/latest/dev/writing-with-agents.html
• https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/Su
bscriptions.html
• https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/M
onitoringLogData.html