Deep Dive- Log analytics with Amazon Elasticsearch Service - AWS Summit Tel Aviv 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Kobi Biton, Solutions Architect, AWS
Ronny Lehmann, CTO, Loom
June 21, 2017
Operational Insights Out Of Your Logs
Automate Track

• Why do we analyze logs?
• Log analysis challenges and solution
• Architecture patterns
• Loom, AI log analysis
• Summary and key takeaways
What to expect from today

• Operational health, Identify service failures
• Near real time Insights & alerts
• Auto Recovery on failure
• Historical Analysis, discover new Insights
Why do we analyze logs?

• High volume, high velocity, scale
• Multiple log sources / data structures
• Act on log events as they enter the
pipeline
• What should we visualize?
Log analysis challenges

Why do we do we a terabyte of logs?

{ "eventVersion": "1.05", "userIdentity": { "type": "IAMUser", "principalId":
"AIDACKCEVSQ6C2EXAMPLE", "arn":"arn:aws:iam::111122223333:user/moshe",
"accountId":"111122223333", "userName": ”moshe" }, "eventTime": "2014-07-08T17:35:27Z",
"eventSource": "signin.amazonaws.com", "eventName": "ConsoleLogin", "awsRegion":"us-east-2",
"sourceIPAddress": ”xxx.xxx.xxx.xxx", "userAgent": "Mozilla/5.0 (Windows NT 6.1; WOW64;
rv:24.0) Gecko/20100101 Firefox/24.0", "errorMessage": "Failedauthentication",
"requestParameters": null, "responseElements": { "ConsoleLogin": "Failure" },
"additionalEventData": { "MobileVersion": "No", "LoginTo":"https://console.aws.amazon.com/sns",
"MFAUsed": "No" }, "eventID": "11ea990b-4678-4bcd-8fbe-62509088b7cf" }
We want to turn this..

{ "eventVersion": "1.05", "userIdentity": { "type": "IAMUser", "principalId":
"AIDACKCEVSQ6C2EXAMPLE", "arn":"arn:aws:iam::111122223333:user/moshe",
"accountId":"111122223333", "userName": ”moshe" }, "eventTime": "2014-07-08T17:35:27Z",
"eventSource": "signin.amazonaws.com", "eventName": "ConsoleLogin", "awsRegion":"us-east-2",
"sourceIPAddress": ”xxx.xxx.xxx.xx", "userAgent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0)
Gecko/20100101 Firefox/24.0", "errorMessage": "Failedauthentication", "requestParameters":
null, "responseElements": { "ConsoleLogin": "Failure" },
"additionalEventData": { "MobileVersion": "No", "LoginTo":"https://console.aws.amazon.com/sns",
"MFAUsed": "No" }, "eventID": "11ea990b-4678-4bcd-8fbe-62509088b7cf" }
Into this…

And then visualize it

Infrastructure Log Sources
• AWS CloudTrail
• VPC flow logs
• Amazon EC2 OS logs

• Amazon S3 bucket server access logs
• Amazon Cloudfront access logs
• Elastic Load Balancer access logs
• Custom EC2 application logs (apache, tomcat)
Application/Service log sources

• Amazon EC2 / AWS Elastic Load Balancer
• Amazon Kinesis Firehose
• Amazon CloudWatch Logs
• Amazon Elasticsearch Service
• Amazon Simple Storage Service (Amazon S3)
AWS Services to address the challenge

Amazon
Kinesis
Firehose
Capture, transform, and load streaming data
into Amazon Kinesis Analytics, Amazon
S3, Amazon Redshift, and Amazon Elasticsearch
Service, enabling near real-time analytics. It is a
fully managed service that automatically scales to
match the throughput of your data and requires no
ongoing administration.

Amazon
CloudWatch
Logs
CloudWatch Logs is a highly available, scalable,
durable, and secure service to manage your
operating system and application log files. It allows
you to ingest, store, filter, search, and archive the
logs.

Amazon
Elasticsearch
Service
Amazon Elasticsearch Service is a cost-effective
managed service that makes it easy to deploy,
manage, and scale open source Elasticsearch for
log analytics, full-text search and more.

Launching an Amazon
Elasticsearch Cluster

Log Analysis architecture patterns

Amazon Elasticsearch
Service
Store, Analyze
Kibana
Visualize
Amazon EC2
Collect, Batch
Direct shipping via Logstash

Amazon Kinesis Firehose
Capture,Transform,Batch,Ship
Service
Store, Analyze
Kibana
Visualize
Decouple using Kinesis Firehose
Amazon EC2
Collect, Batch
Kinesis Firehose Agent
Built-In destination

Service
Store, Analyze
Kibana
Visualize
Amazon EC2
Collect, Batch
AWS Lambda
Transform

Service
Store, Analyze
Kibana
Visualize
Amazon EC2
Collect, Batch
AWS Lambda
Transform
Amazon S3
Durable Long Term Storage

Decouple using S3 / CloudWatch Logs
AWS
Lambda
Amazon S3
bucket
CloudWatch
Log Group
ELB Logs CloudFront
Access Logs
VPC Flow
Logs
CloudTrail
Amazon ES
Metric Filter Amazon
CloudWatch
Alarm
EC2 OS Logs

AWS Log Analysis Partner Eco-System

Ronny Lehmann, CTO
June 21, 2017

Artificial Intelligence to
Predict & Prevent Problems in
your production environment

Easy with AWS:
✓ Collecting
✓ Storing
✓ Querying
✓ Visualizing
What’s the Bottleneck?
You are.

Structure & Contextualize
Continuously Analyze
Triage
Fix
Why so Manual?

Humans
Good at Top-Down tasks
• Deep reasoning
• Contextual thinking
• Tired
• Bored
• Lazy
• Frustrated
• Married
• Real-time trend
detection
• Pattern Recognition
• Large Dimensionality
• Complex State
• Strict Methodology
Machines
Superior at bottom-up tasks

“I’ve been hearing this for 20 years”
Total Recall, a movie based on
a book from 1966, featuring a
self-driving car as science
fiction.
If Artificial-Intelligence has
matured enough to drive your
car, it can probably also help
with your IT.

Metric Collectors
Metrics
2.0
Log Collectors
RSYSLOG
Processing
Analysis
Alerting
Receive
Process
Anomaly
Detection
Root
Cause
Analysis
Insight

Sep 27 14:25:54 acme sshd[749]: WARN - Failed password for user ronny from 192.168.118.1 port 48278 ssh2
Recover the generating-pattern for the event:
Automatically transform every event to a collection of metrics:
Server megatron Histogram
Service sshd Meter
User ronny Meter
source_IP 192.168.118.1 Meter
Severity Warn Meter
Keyword Failed Meter
Pattern 16225 Meter
Failed password for user [user] from [source_IP] port [source_port] ssh2

Sep 27 14:25:54 megatron sshd[7498]: WARN - Failed password for user ronny from 192.168.118.1 port…
Sep 27 14:25:54 megatron sshd[7498]: WARN - Failed password for user dror from 192.168.118.1 port…
Sep 27 14:25:54 megatron sshd[7498]: WARN - Failed password for user john from 192.168.118.1 port…
Sep 27 14:25:55 megatron sshd[7498]: WARN - Failed password for user mike from 192.168.118.1 port…
Sep 27 14:25:55 megatron sshd[7498]: WARN - Failed password for user orr from 192.168.118.1 port…
Sep 27 14:25:55 megatron sshd[7498]: WARN - Failed password for user george from 192.168.118.1 port…
Sep 27 14:25:55 megatron sshd[7498]: WARN - Failed password for user harris from 192.168.118.1 port…
Sep 27 14:25:56 megatron sshd[7498]: WARN - Failed password for user paul from 192.168.118.1 port…
Sep 27 14:25:56 megatron sshd[7498]: WARN - Failed password for user noam from 192.168.118.1 port…
Sep 27 14:25:56 megatron sshd[7498]: WARN - Failed password for user ringo from 192.168.118.1 port…
Sep 27 14:25:56 megatron sshd[7498]: WARN - Failed password for user eli from 192.168.118.1 port…
Sep 27 14:25:56 megatron sshd[7498]: WARN - Failed password for user keren from 192.168.118.1 port…
Sep 27 14:25:57 megatron sshd[7498]: WARN - Failed password for user larissa from 192.168.118.1 port…

Sep 27 14:25:54 megatron sshd[7498]: WARN - Failed password for user dror from 192.168.118.1 port…
Sep 27 14:25:54 megatron sshd[7498]: WARN - Failed password for user john from 192.168.118.1 port…
Sep 27 14:25:55 megatron sshd[7498]: WARN - Failed password for user mike from 192.168.118.1 port…
Sep 27 14:25:55 megatron sshd[7498]: WARN - Failed password for user orr from 192.168.118.1 port…
Sep 27 14:25:55 megatron sshd[7498]: WARN - Failed password for user george from 192.168.118.1 port…
Sep 27 14:25:55 megatron sshd[7498]: WARN - Failed password for user harris from 192.168.118.1 port…
Sep 27 14:25:56 megatron sshd[7498]: WARN - Failed password for user paul from 192.168.118.1 port…
Sep 27 14:25:56 megatron sshd[7498]: WARN - Failed password for user noam from 192.168.118.1 port…
Sep 27 14:25:56 megatron sshd[7498]: WARN - Failed password for user ringo from 192.168.118.1 port…
Sep 27 14:25:56 megatron sshd[7498]: WARN - Failed password for user eli from 192.168.118.1 port…
Sep 27 14:25:56 megatron sshd[7498]: WARN - Failed password for user keren from 192.168.118.1 port…
Sep 27 14:25:57 megatron sshd[7498]: WARN - Failed password for user larissa from 192.168.118.1 port…
ronny is mentioned more than normal in the context of ssh failures
The context of ssh failures is mentioned more than normal

Summary & Key Takeaways
• Collect Logs
• Define clear requirements
• Learn and experiment
Ø Open-Source tools and technologies
Ø AWS Managed services
Ø AWS Partner Eco-System

Useful Links
• AWS Centralized logging solution
https://aws.amazon.com/answers/logging/centralized-logging/
• Real-time Processing of Log Data with Subscriptions
https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/
Subscriptions.html
• Amazon Kinesis Firehose Data Transformation with AWS Lambda
https://aws.amazon.com/blogs/compute/amazon-kinesis-firehose-data-
transformation-with-aws-lambda/
• Elasticsearch 5.3 now available on Amazon Elasticsearch Service
https://aws.amazon.com/about-aws/whats-new/2017/06/elasticsearch-5-3-
now-available-on-amazon-elasticsearch-service/

Deep Dive- Log analytics with Amazon Elasticsearch Service - AWS Summit Tel Aviv 2017

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Deep Dive- Log analytics with Amazon Elasticsearch Service - AWS Summit Tel Aviv 2017

Similar to Deep Dive- Log analytics with Amazon Elasticsearch Service - AWS Summit Tel Aviv 2017 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Deep Dive- Log analytics with Amazon Elasticsearch Service - AWS Summit Tel Aviv 2017