- Applications generate logs. Infrastructure generates logs. Even humans generate logs (though we usually call that “medical data”). By ingesting and analyzing logs, you can gain understanding of how complex systems operate and quickly discover and diagnose when they don’t work as they should. In this workshop, we ingest and analyze log streams using Amazon Kinesis Firehose and Amazon Elasticsearch Service. You should come with an understanding of AWS fundamentals (Amazon EC2, Amazon S3, and security groups). You need a laptop with a Chrome or Firefox browser.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Easy and Scalable Log Analytics with
Amazon Elasticsearch Service
J o n H a n d l e r , P r i n c i p a l S A
K e v i n F a l l i s , S A
D a r i n B r i s k m a n , E v a n g e l i s t

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Overview
• Build and deploy a
web app to to search a
small movies database
• Deploy an Amazon
Elasticsearch Service
domain in your VPC
• Monitor the web app
by sending Apache
access_logs to
Amazon Elasticsearch
Service

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Transition from IT
to DevOps
Increase in IoT and
Mobile Devices
Cloud-based
architectures
Machine-generated data is growing 10x faster than business data
Source: insideBigData - The Exponential Growth of Data, February 16, 2017
THE EXPLOSION OF MACHINE-GENERATED DATA

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Source: TechCrunch survey of popular open source software from April’17
Open source
Fast time to value
Easy ingestion
Easy visualization
High performance and distributed
Best analytics and search
LOG ANALYTICS FUELING ELASTICSEARCH GROWTH
Rank Project Name
Overall Project
Rating
1 Linux 100.00
2 Git 31.10
3 MySQL 25.23
4 Node.js 22.75
5 Docker 22.61
6 Hadoop 16.19
7 Elasticsearch 15.72
8 Spark 14.99
9 MongoDB 14.68
10 Selenium 12.81
11 NPM 12.31
12 Redis 11.61
B E N E F I T S

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ELASTICSEARCH LEADING USE CASES
Application Monitoring &
Root-cause Analysis
Security Information and Event
Management (SIEM)
IoT & Mobile Business & Clickstream Analytics
Provides developers with a high
performance, self-service operational
monitoring and analytics platform
Enables security practitioners to centralize
and analyze events from across the entire
organization
Gives developers and lines of business
users real-time location-aware insights
into their device fleets
Provides business users with a real-time view
of the performance of their web content and
e-commerce platforms

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Elasticsearch Service is a
fully managed service that makes
it easy to deploy, manage, and
scale Elasticsearch and Kibana
A M A Z ON E L A ST IC SE AR C H S E R V IC E
+

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application Monitoring & Root-cause Analysis
C A S E S T U D Y : EXPEDI A
Logs, lots and lots of logs. How to cost effectively monitor logs?
Require centralized logging infrastructure
Did not have the man power to manage infrastructure
P R O B L E M
Quick insights: Able to identify and troubleshoot issues in real-time
Secure: Integrated w/ AWS IAM
Scalable: Cluster sizes are able to grow to accommodate additional log sources
B E N E F I T S
Streaming CloudTrail logs, application logs, and Docker startup logs
to Elasticsearch
Created centralized logging service for all team members
Using Kibana for visualizations and for ES queries
S O L U T I O N
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Business and Clickstream Analytics
C A S E S T U D Y : FI NANC I AL T I MES
What stories do our readers care about? What’s hot?
Required a custom clickstream analytics solution
Need a solution that delivers analytics in real-time
Did not have a team to manage analytics infrastructure
P R O B L E M
B E N E F I T S
Streaming user data to Elasticsearch Service for analysis. Created
their own custom dashboards for editors and journalists – Lantern.
Lantern - ”shines a light” on reader activity for the editors and
journalists at the FT
Critical tool for making editorial decisions. Daily editorial meetings
start by looking at Lantern dashboard
S O L U T I O N
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reliability : Lantern is used throughout the day by journalists and editors. Relying on Amazon to manage
their systems for maximum uptime.
Cost savings: Able to easily tune their cluster to meet their needs with minimal management overhead

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
B E N E F I T S OF AMAZON ELAST I C SEAR C H SER V I C E
Supports Open-Source
APIs and Tools
Drop-in replacement with no
need to learn new APIs or skills
Easy to Use
Deploy a production-ready
Elasticsearch cluster in
minutes
Scalable
Resize your cluster with a few
clicks or a single API call
Secure
Deploy into your VPC and restrict
access using security groups and
IAM policies
Highly Available
Replicate across Availability
Zones, with monitoring and
automated self-healing
Tightly Integrated with
Other AWS Services
Seamless data ingestion, security,
auditing and orchestration

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Software & Internet Financial ServicesEducation Technology BioTech and Pharma
Media and Entertainment Social Media Telecommunications Travel & Transportation
Real Estate Logistics & Operations Publishing Other
AMAZON ELASTI CSEARCH SERV I CE CUSTOMERS

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Elastic Stack is the foundation for the solution
Elasticsearch – distributed search engine for logs
Logstash – simple tool for transforming and streaming data into ES
Kibana – Easy to use tool for visualization of log data in Elasticsearch

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
* NEW * Amazon Elasticsearch VPC Support
• Private networking between
your VPC and Amazon
Elasticsearch Service
• Traffic does not
traverse the public
internet
• Use IAM policies and
security groups for
authentication and access
control
VPC subnet
security group
VPC subnet
security group
Amazon Elasticsearch Service
Data Master
Data
Master
IAM
IAM
Availability
Zone B
Availability
Zone A

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Architectural Approach (Logging)
FB
Apache
FB
Apache
FB
Apache
Log Shippers Buffer Log Indexers Search
ApplicationTraffic
Amazon Elasticsearch Service

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Architectural Approach (Web Serving)
App
Apache
App
Apache
App
Apache
Application
PHP/Apache
Search
Amazon Elasticsearch Service

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Portal
SSH
Secure Access
Analysis & Visualization
Amazon Elasticsearch Service
Architectural Approach (Analysis)

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Managed services in the lab
• Amazon Elasticsearch Service – Analysis and search for
log data
• Elasticache (Redis) – Buffering and collation for logs
flowing through the system
• Route 53 – (DNS) – DNS name resolution to route traffic
to Redis and Amazon Elasticsearch Service
• CFN – (CloudFormation) – Deploys the full solution from
the AWS console
• CloudTrail – Audit access to the administrative
commands across AWS

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Security for the lab architecture
• VPC – (Virtual Private Cloud) - Isolated networking that
keeps traffic within the AWS network
• ENI – (Elastic Network Interface) – Private networking for
managed services – Elasticache & Elasticsearch
• NAT Gateway – One IP address that hides my servers
from the world
• ALB – (Application Load Balancer) – Limits access and
exposes perimeter to DDOS attacks
• Security Groups – ensure only trusted entities can talk to
my servers
• Subnets – Coupled with Route Tables Limits Specific
Traffic
SG

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Assembly – Network Stack
Proxy
Admin /
Security
YUM
Public
Subnet
Public
Subnet
AZ 1
AZ 2
NAT Gateway
Bastion
Internet
Gateway
Private Subnet
Private Subnet

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Assembly – Managed Services
Proxy
Admin /
Security
YUM
Private Subnet
Private Subnet
Public
Subnet
Public
Subnet
AZ 1
AZ 2
NAT Gateway
Bastion
Internet
Gateway
Redis ElasticsearchENI

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ENI Logstash Redis ElasticsearchProxy
Assembly – Logstash Stack
Admin /
Security
Users
YUM

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Assembly – Web Services / Filebeat
ENI LogstashApache / Filebeat Redis ElasticsearchProxy
Admin /
Security
YUM
Users

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cloudformation is the Deployment Tool

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Templates
• Networking – creates the VPC, subnets, and portal
• Redis – creates the Elasticache Redis cluster in the VPC
• Logstash – creates instances for and deploys Logstash
• Application – deploys the application and filebeats

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Understand Kibana

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
host:199.72.81.55 with <histogram of verb>
1,
4,
8,
12,
30,
42,
58,
100
...
Look up
199.72.81.55
Field data
GET
GET
POST
GET
PUT
GET
GET
POST
Buckets
GET
POST
PUT
5
2
1
Counts

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon ES aggregations
Buckets – a collection of documents meeting some
criterion
Metrics – calculations on the content of buckets
Bucket: time
Metric:count

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A more complicated aggregation
Bucket: ARN
Bucket: Region
Bucket: eventName
Metric: Count

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon ES: Metrics for summary reporting

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon ES: Line charts graphing metrics

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon ES: Area charts break out
components of a total

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon ES: Pie charts show correlations

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon ES: Heatmaps to find hot spots

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon ES: Vertical bar charts for histograms

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!
C L I C K T O A D D T E X T
C L I C K T O A D D T E X T