This document provides an overview of using Amazon Elasticsearch Service for log analytics. It describes how to ingest logs from data sources using Amazon Kinesis Firehose into an Amazon Elasticsearch Service cluster. It then discusses how to analyze the logs stored in Elasticsearch using aggregations and visualizing them in Kibana. It provides best practices for configuring Elasticsearch clusters on AWS and optimizing them for ingest and analytics workloads.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jon Handler, AWS Principal Solutions Architect,
Search Services
April 19, 2017
Deep Dive: Log Analytics with
Amazon Elasticsearch Service
BDA402

2. What to do with a terabyte of logs?

3. Visualize it with Kibana!

4. data source Amazon Kinesis Firehose Amazon Elasticsearch
Service
Kibana
Log analytics architecture

5. Amazon Elasticsearch Service is a cost-effective
managed service that makes it easy to deploy,
manage, and scale open source Elasticsearch for log
analytics, full-text search, and more.
Amazon
Elasticsearch
Service

6. Amazon Elasticsearch Service benefits
Easy to use
Open-source
compatible
Secure
Highly available
AWS integrated
Scalable

7. Leading enterprises trust Amazon Elasticsearch
Service for their search and analytics applications
Media &
Entertainment
Online
Services
Technology Other

8. Use Case: Adobe Developer Platform (Adobe I/O)
P R O B L E M
• Cost-effective
monitoring for XL
amount of log data
• 200,000+ API calls per
second at peak -
destinations, response
times, bandwidth
• Integrate seamlessly
with other components
of AWS ecosystem
B E N E F I T S
• Management and
operational simplicity
• Flexibility to try out
different cluster config
during dev and test
Amazon
Kinesis Streams
Spark Streaming on
Amazon EMR
Amazon
Elasticsearch Service
Data
Sources
1
S O L U T I O N
• Log data is sent via
Amazon Kinesis
Streams to Amazon
Elasticsearch Service,
then displayed using
AES Kibana
• Adobe team can easily
see traffic patterns and
error rates, and quickly
identify anomalies and
potential issues

9. Get set up right

10. Amazon ES overview
Amazon Route
53
Elastic Load
Balancing
IAM
CloudWatch
Elasticsearch API
CloudTrail

12. Data pattern
Amazon ES cluster
logs_01.21.2017
logs_01.22.2017
logs_01.23.2017
logs_01.24.2017
logs_01.25.2017
logs_01.26.2017
logs_01.27.2017
Shard 1
Shard 2
Shard 3
host
ident
auth
timestamp
etc.
Each index has
multiple shards
Each shard contains
a set of documents
Each document contains
a set of fields and values
One index per day

13. Deployment of indices to a cluster
• Index 1
– Shard 1
– Shard 2
– Shard 3
• Index 2
– Shard 1
– Shard 2
– Shard 3
Amazon ES cluster
1
2
3
1
2
3
1
2
3
1
2
3
Primary Replica
1
3
3
1
Instance 1,
Master
2
1
1
2
Instance 2
3
2
2
3
Instance 3

15. How many instances?
The index size will be about the same as the
corpus of source documents
• Double this if you are deploying an index replica
Size based on storage requirements
• Either local storage or up to 1.5 TB of Amazon Elastic
Block Store (EBS) per instance
• Example: 2 TB corpus will need 4 instances
– Assuming a replica and using EBS
– With i2.2xlarge nodes using 1.6 TB ephemeral storage

17. Determining instance type
Instance type is workload-dependent
• T2: dev, test, QA
• M3/M4: solid performance
• R3/R4: heavier queries, aggregations
• C4: high throughput query loads
• I2: largest storage option

19. Cluster with no dedicated masters
• Every cluster has a master
• The master orchestrates
cluster operations – shard
locations and schema
• Problem: if the master
function is assigned to a
data instance, that instance
can become unavailable
from query/ingest load
Amazon ES cluster
1
3
3
1
Instance 1,
Master
2
1
1
2
Instance 2
3
2
2
3
Instance 3

20. Cluster with dedicated masters
Amazon ES cluster
1
3
3
1
Instance 1
2
1
1
2
Instance 2
3
2
2
3
Instance 3Dedicated master nodes
Data nodes: queries and updates

21. Master node recommendations
• Masters do not participate in query or ingest processing
• Because the do less, they can be smaller than the data instances
• Master election is based on a quorum of eligible nodes, you need an odd
number to protect against network splits. One is too few, so choose 3 at
least.
Number of data nodes Master node instance type
< 10 m3.medium+, c4.large+
< 20 m3/4.large+, r3/4.large+
< 50 c4.xlarge+, m3/4.xlarge+, r4.xlarge+
Up to 100 m4.xlarge+, r4.xlarge+, m3.xlarge+,
c4.2xlarge+, r3.xlarge+, i2.xlarge+

23. Cluster with zone awareness
Amazon ES cluster
1
3
Instance 1
2
1 2
Instance 2
3
2
1
Instance 3
Availability Zone 1 Availability Zone 2
2
1
Instance 4
3
3

24. Architecture for small use cases
• Logstash co-located on the
Application instance
• SigV4 signing via provided
output plugin
• Up to 200 GB of data
• m3.medium + 100G EBS
data nodes
• 3x m3.medium master nodes
Application
Instance

25. Architecture for large use cases
Amazon
DynamoDB
AWS
Lambda
Amazon S3
bucket
Amazon
CloudWatch
• Data flows from instances
and applications via
Lambda; CWL is implicit
• SigV4 signing via
Lambda/roles
• Up to 5 TB of data
• r3.2xlarge + 512 GB EBS
data nodes
• 3x m3.medium master nodes

26. Architecture for XL use cases
Amazon
Kinesis
• Ingest supported through
high-volume technologies
like Spark or Kinesis
• Up to 60 TB of data today
• R3.8xlarge + 640 GB data
nodes
• 3x m3.xlarge master nodes
Amazon
EMR

27. Best practices
• Data nodes = Storage needed/Storage per node
• Use GP2 EBS volumes
• Use 3 dedicated master nodes for production
deployments
• Enable zone awareness
• Set indices.fielddata.cache.size = 40

28. Amazon Kinesis Firehose

29. Amazon Kinesis Firehose overview
Delivery Stream: Underlying
AWS resource
Destination: Amazon ES,
Amazon Redshift, Amazon S3,
or Amazon Kinesis Analytics
Record: Put records in
streams to deliver to
destinations

30. Kinesis Firehose delivery architecture with
transformations
intermediate
Amazon S3
bucket
backup S3 bucket
source records
data source
source records
Amazon Elasticsearch
Service
Firehose
delivery stream
transformed
records transformed
records
transformation failure
delivery failure

31. Kinesis Firehose features for ingest
Serverless scaling Error handling S3 Backup

32. Best practices
• Use smaller buffer sizes to increase throughput, but be
careful of concurrency
• Use index rotation based on sizing
• Default stream limits: 2,000 transactions/second, 5,000
records/second, and 5 MB/second

33. Log Analytics with
Aggregations

34. Amazon ES aggregations
Buckets – a collection of documents meeting some criterion
Metrics – calculations on the content of buckets
Bucket: time
Metric:count

35. host:199.72.81.55 with <histogram of verb>
1,
4,
8,
12,
30,
42,
58,
100
...
Look up
199.72.81.55
Field data
GET
GET
POST
GET
PUT
GET
GET
POST
Buckets
GET
POST
PUT
5
2
1
Counts

36. A more complicated aggregation
Bucket: ARN
Bucket: Region
Bucket: eventName
Metric: Count

37. Best practices
• Elasticsearch provides statistical evaluations based on
field data gathered from matching documents
• Visualizations are based on buckets/metrics
• Use a histogram on the x-axis first, then sub-aggregate

38. • Easily run Elasticsearch on AWS Cloud using
Amazon Elasticsearch Service
• Use Amazon Kinesis Firehose to ingest data
• Use Kibana for monitoring, Elasticsearch
queries for deeper analysisAmazon
Elasticsearch
Service

39. What to do next
Qwiklab:
https://qwiklabs.com/searches/lab?keywords=introduction
%20to%20amazon%20elasticsearch%20service
Centralized logging solution
https://aws.amazon.com/answers/logging/centralized-
logging/
Our overview page on AWS
https://aws.amazon.com/elasticsearch-service/

40. Thank you!