Everything generates logs. Applications, infrastructure, security ... everything. Keeping track of the flood of log data is a big challenge, yet critical to your ability to understand your systems and troubleshoot (or prevent) issues. In this session, we will use both Amazon CloudWatch and application logs to show you how to build an end-to-end log analytics solution. First, we cover how to configure an Amazon Elaticsearch Service domain and ingest data into it using Amazon Kinesis Firehose, demonstrating how easy it is to transform data with Firehose. We look at best practices for choosing instance types, storage options, shard counts, and index rotations based on the throughput of incoming data and configure a secure analytics environment. We demonstrate how to set up a Kibana dashboard and build custom dashboard widgets. Finally, we dive deep into the Elasticsearch query DSL and review approaches for generating custom, ad-hoc reports.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jon Handler, Principal Solutions Architect, AWS
July 27, 2017
Deep Dive – Log Analytics with
Amazon Elasticsearch Service

2. What produces data?
• Metering
Records
• Mobile
Apps
• IoT
Sensors
Web
Clickstream
• Enterprise
documents
• Application
Logs
[Wed Oct 11 14:32:52
2000] [error] [client
127.0.0.1] client
denied by server
configuration:
/export/home/live/ap/ht
docs/test

3. The problem with all of that data

4. The solution

5. Amazon Elasticsearch Service is a cost-effective
managed service that makes it easy to deploy,
manage, and scale open-source Elasticsearch for log
analytics, full-text search, and more.
Amazon
Elasticsearch
Service

6. Data source /
Kinesis
Firehose Agent
Amazon Kinesis Firehose Amazon Elasticsearch
Service
Kibana
Log analytics architecture

7. Easy to Use
Deploy a production-ready Elasticsearch
cluster in minutes
Simplifies time-consuming management
tasks such as software patching, failure
recovery, backups, and monitoring
Open
Get direct access to the Elasticsearch
open-source API
Fully compatible with the open-source
Elasticsearch API, for all code and
applications
Secure
Secure Elasticsearch clusters with AWS
Identity and Access Management (IAM)
policies with fine-grained access control
access for users and endpoints
Automatically applies security patches
without disruption, keeping Elasticsearch
environments secure
Available
Provides high availability using Zone
Awareness, which replicates data between
two Availability Zones
Monitors the health of clusters and
automatically replaces failed nodes,
without service disruption
AWS Integrated
Integrates with Amazon Kinesis Firehose,
AWS IoT, and Amazon CloudWatch Logs for
seamless data ingestion
AWS CloudTrail for auditing, AWS Identity
and Access Management (IAM) for
security, and AWS CloudFormation for
cloud orchestration
Scalable
Scale clusters from a single node up to 20
nodes
Configure clusters to meet performance
requirements by selecting from a range of
instance types and storage options,
including SSD-powered EBS volumes
Amazon Elasticsearch Service Benefits

8. Amazon Elasticsearch Service leading use cases
Log Analytics &
Operational Monitoring
• Monitor the performance of
applications, web servers, and
hardware
• Easy to use, powerful data
visualization tools to detect
issues quickly
• Dig into logs in an intuitive,
fine-grained way
• Kibana provides fast, easy
visualization
Search
• Application or website provides
search capabilities over diverse
documents
• Tasked with making this knowledge
base searchable and accessible
• Text matching, faceting, filtering,
fuzzy search, autocomplete,
highlighting, and other search
features
• Query API to support application
search

9. Leading enterprises trust Amazon Elasticsearch
Service for their search and analytics applications
Media &
Entertainment
Online
Services
Technology Other

10. Adobe Developer Platform (Adobe I/O)
P R O B L E M
• Cost-effective monitor
for XL amount of log
data
• Over 200,000 API calls
per second at peak -
destinations, response
times, bandwidth
• Integrate seamlessly
with other components
of AWS ecosystem
S O L U T I O N
• Log data is routed
with Amazon Kinesis
to Amazon
Elasticsearch Service,
then displayed using
AES Kibana
• Adobe team can
easily see traffic
patterns and error
rates, quickly
identifying anomalies
and potential
challenges
B E N E F I T S
• Management and
operational simplicity
• Flexibility to try out
different cluster config
during dev and test
Amazon
Kinesis
Streams
Spark Streaming
Amazon
Elasticsearch
Service
Data
Sources
1

11. McGraw Hill Education
P R O B L E M
• Supporting a wide catalog
across multiple services in
multiple jurisdictions
• Over 100 million learning
events each month
• Tests, quizzes, learning
modules begun / completed
/ abandoned
S O L U T I O N
• Search and analyze test
results, student/teacher
interaction, teacher
effectiveness, student
progress
• Analytics of applications
and infrastructure are now
integrated to understand
operations in real time
B E N E F I T S
• Confidence to scale
throughout the school year.
From 0 to 32 TB in 9 months
• Focus on their business, not
their infrastructure

12. Get set up right

13. Easy to use and scalable
AWS SDK
AWS CLI
AWS
CloudFormation
Elasticsearch
data nodes
Elasticsearch
master nodes
Elastic Load
Balancing
AWS IAM
CloudWatchCloudTrail
Amazon Elasticsearch Service domain

16. Data pattern
Amazon ES cluster
logs_01.21.2017
logs_01.22.2017
logs_01.23.2017
logs_01.24.2017
logs_01.25.2017
logs_01.26.2017
logs_01.27.2017
Shard 1
Shard 2
Shard 3
host
ident
auth
timestamp
etc.
Each index has
multiple shards
Each shard contains
a set of documents
Each document contains
a set of fields and values
One index per day

17. Deployment of indices to a cluster
• Index 1
– Shard 1
– Shard 2
– Shard 3
• Index 2
– Shard 1
– Shard 2
– Shard 3
Amazon ES cluster
1
2
3
1
2
3
1
2
3
1
2
3
Primary Replica
1
3
3
1
Instance 1,
Master
2
1
1
2
Instance 2
3
2
2
3
Instance 3

18. How many instances?
The index size will be about the same as the
corpus of source documents
• Double this if you are deploying an index replica
Size based on storage requirements
• Either local storage or up to 1.5 TB of Amazon Elastic
Block Store (EBS) per instance
• Example: 2 TB corpus will need 4 instances
– Assuming a replica and using EBS
– With i2.2xlarge nodes using 1.6 TB ephemeral storage

20. Instance type recommendations
Instance Workload
T2 Entry point. Dev and test. OK for dedicated masters.
M3, M4 Equal read and write volumes.
R3, R4 Read-heavy or workloads with high memory demands (e.g.,
aggregations).
C4 High concurrency/indexing workloads
I2 Up to 1.6 TB of SSD instance storage.

22. Cluster with no dedicated masters
Amazon ES cluster
1
3
3
1
Instance 1,
Master
2
1
1
2
Instance 2
3
2
2
3
Instance 3

23. Cluster with dedicated masters
Amazon ES cluster
1
3
3
1
Instance 1
2
1
1
2
Instance 2
3
2
2
3
Instance 3Dedicated master nodes
Data nodes: queries and updates

24. Master node recommendations
Number of data nodes Master node instance type
< 10 m3.medium+
< 20 m4.large+
<= 50 c4.xlarge+
50-100 c4.2xlarge+
Always use an odd number of masters, >= 3

26. Cluster with zone awareness
Amazon ES cluster
1
3
Instance 1
2
1 2
Instance 2
3
2
1
Instance 3
Availability Zone 1 Availability Zone 2
2
1
Instance 4
3
3

27. Small use cases
• Logstash co-located on the
Application instance
• SigV4 signing via provided
output plugin
• Up to 200 GB of data
• m3.medium + 100G EBS
data nodes
• 3x m3.medium master nodes
Application
Instance

28. Large use cases
Amazon
DynamoDB
AWS
Lambda
Amazon S3
bucket
Amazon
CloudWatch
• Data flows from instances
and applications via
Lambda; CWL is implicit
• SigV4 signing via
Lambda/roles
• Up to 5 TB of data
• r3.2xlarge + 512 GB EBS
data nodes
• 3x m3.medium master nodes

29. XL use cases
Amazon
Kinesis
• Ingest supported through
high-volume technologies
like Spark or Kinesis
• Up to 60 TB of data today
• R3.8xlarge + 640GB data
nodes
• 3x m3.xlarge master nodes
Amazon
EMR

30. Best practices
Data nodes = Storage needed/Storage per node
Use GP2 EBS volumes
Use 3 dedicated master nodes for production deployments
Enable Zone Awareness
Set indices.fielddata.cache.size = 40

31. Kinesis Firehose

32. Kinesis Firehose overview
Delivery Stream: Underlying
AWS resource
Destination: Amazon ES,
Amazon Redshift, or Amazon
S3
Record: Put records in
streams to deliver to
destinations

33. Kinesis Firehose delivery architecture with
transformations
S3 bucket
source records
data source
source records
Amazon Elasticsearch
Service
Firehose
delivery stream
transformed
records
delivery failure
Data transformation
function
transformation failure

34. Lambda blueprints for common use cases

35. Transform this
199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] "GET /history/apollo/ HTTP/1.0" 200 6245
To this
{
"verb": "GET",
"ident": "-",
"bytes": 6245,
"@timestamp": "1995-07-01T00:00:01",
"request": "GET /history/apollo/ HTTP/1.0",
"host": "199.72.81.55",
"authuser": "-",
"@timestamp_utc": "1995-07-01T04:00:01+00:00",
"timezone": "-0400",
"response": 200
}

36. Kinesis Firehose features for ingest
Serverless scale Error handling S3 Backup

37. Demo

38. Best practices
Use smaller buffer sizes to increase throughput, but be
careful of concurrency
Use index rotation based on sizing
Default: stream limits: 2,000 transactions/second, 5,000
records/second, and 5 MB/second

39. Log analysis with aggregations

40. host:199.72.81.55 with <histogram of verb>
1,
4,
8,
12,
30,
42,
58,
100
...
Look up
199.72.81.55
Field data
GET
GET
POST
GET
PUT
GET
GET
POST
Buckets
GET
POST
PUT
5
2
1
Counts

41. Amazon ES aggregations
Buckets – a collection of documents meeting some criterion
Metrics – calculations on the content of buckets
Bucket: time
Metric:count

42. A more complicated aggregation
Bucket: ARN
Bucket: Region
Bucket: eventName
Metric: Count

43. Demo

44. Best practices
Elasticsearch provides statistical evaluations based on field
data gathered from matching documents
Visualizations are based on buckets/metrics
Use a histogram on the x-axis first, then sub-aggregate

45. Run Elasticsearch in the AWS cloud with Amazon
Elasticsearch Service
Use Kinesis Firehose to ingest data simply
Kibana for monitoring, Elasticsearch queries for
deeper analysisAmazon
Elasticsearch
Service

46. What to do next
Qwiklab:
https://qwiklabs.com/searches/lab?keywords=introduction%20to%20a
mazon%20elasticsearch%20service
Centralized logging solution
https://aws.amazon.com/answers/logging/centralized-logging/
Our overview page on AWS
https://aws.amazon.com/elasticsearch-service/
Questions? Contact me at handler@amazon.com

47. Thank you!