Analyzing your web and application logs on AWS. Utrecht AWS Dev Day

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2019.06.18
U T R E C H T

Analyzing your web and application logs
Athena, Cloud Watch Logs, Lambda, Kinesis Firehose, and the Elasticsearch Service
Javier Ramirez
AWS Tech Evangelist
@supercoco9
B A R 2
U T R E C H T
2019.06.18

A day in the life of an engineer

My journey from beginner to professional developer
Solution
I run all my
programs
interactively
Late 80s
The Amstrad and
Spectrum years
Print before every
command, see
where it breaks
I run some
programs in the
background
Early 90s
The Amiga and PC years
Open a file for
writing and print
to that file
I need to have
different levels of
debugging.
Late 90s
The Solaris years
Fill the code with
IFs, to
conditionally
print to the file
50% of my
code is
If(debug)
Early 00s
The Linux years
Give Log4j a try
My hard
drive is full
Mid 00s
The NFS years
Hello logrotate
I have no
Idea in
which
servers my
logs are
Early 2010s
The Cloud years
Ship to S3
My logs are
full of
insights
Mid 2010s
The Logstash years
Normalise, Enrich,
index, and analyse

Two types of log analytics
• Batch analytics
• Operational (~real-time) analytics

Amazon Global Network
• Redundant 100GbE network
• Redundant private capacity
between all Regions except China
180 Global CloudFront PoPs
(11 regional edge caches)
a e
o
q
i
h
Paris
Sweden
AWS GovCloud East
First 5 years: 4 regions
2016–2020: 14 regions
Next 5 years: 7 regions
A W S
REGIONS
2 1 R e g i o n s 6 6 A Z s
d
m
c
g
b
n
s
k
v
i
i
i
i i
i
i
i
Milan
i
Cape Town

Measure Amazon CloudFront performance
• Third-party tools
• Cedexis
• WebPageTest

Measure Amazon CloudFront performance
• CloudFront Access Logs
• Delivered to an Amazon Simple Storage Service (Amazon S3) bucket
• Includes timeToDownload, cache HIT/MISS, errors, and more
#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query
cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-
cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields
2018-04-07 14:50:37 JFK5 562 207.46.13.70 GET d3vxovn305eucd.cloudfront.net /robots.txt 301
- Mozilla/5.0%2520(compatible;%2520bingbot/2.0;%2520+http://www.bing.com/bingbot.htm) - -
Redirect 9N2U5NMte4dco5qPUGnZM6rRxKsfLtRWbpTZzWHKFsTGxw_Zh2bzfA== dkaran.com http 243 0.000 -
- - Redirect HTTP/1.1 - -
2018-04-07 14:50:37 SEA32 2039 207.46.13.70 GET d3vxovn305eucd.cloudfront.net /robots.txt 404
- Mozilla/5.0%2520(compatible;%2520bingbot/2.0;%2520+http://www.bing.com/bingbot.htm) - -
Error _3CX08QRWugzYNiW0lhiK42DvAfA6BFavZc_Rs4NSpWfvZbB8BrQRQ== dkaran.com
2018-04-07 14:50:57 SEA32 7104 40.77.178.206 GET d3vxovn305eucd.cloudfront.net /style.css 200
https://dkaran.com/
Mozilla/5.0%2520(iPhone;%2520CPU%2520iPhone%2520OS%25207_0%2520like%2520Mac%2520OS%2520X)%2520AppleWebKit/537.51.1%25
20(KHTML,%2520like%2520Gecko)%2520Version/7.0%2520Mobile/11A465%2520Safari/9537.53%2520BingPreview/1.0b- -
Miss 8RDvgIp8Horvy8Lk8vlE_xZk67lmCpdM83z-aXmKi19Gl14scChbow== dkaran.com https 301 0.070 -
TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss HTTP/1.1 - -

Query Amazon CloudFront logs with Amazon Athena
• Enable access logs on your CloudFront distribution
• Logs are delivered to the Amazon S3 bucket of your choice
CloudFront
log files

Athena queries for CloudFront logs
CREATE EXTERNAL TABLE IF NOT EXISTS default.cloudfront_logs (
`date` DATE,
time STRING,
location STRING,
bytes BIGINT,
requestip STRING,
method STRING,
host STRING,
uri STRING,
status INT,
referrer STRING,
useragent STRING,
querystring STRING,
cookie STRING,
resulttype STRING,
requestid STRING,
hostheader STRING,
requestprotocol STRING,
requestbytes BIGINT,
timetaken FLOAT,
xforwardedfor STRING,
sslprotocol STRING,
sslcipher STRING,
responseresulttype STRING,
httpversion STRING,
filestatus STRING,
encryptedfields INT
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY 't'
LOCATION ‘s3://javier-cloud-logs.s3.amazonaws.com/cdn/’
TBLPROPERTIES ( 'skip.header.line.count'='2' )

Athena queries for CloudFront logs
SELECT date,
time,
location,
requestip,
uri,
resulttype,
requestprotocol,
requestbytes,
timetaken
FROM "default"."cloudfront_logs"
WHERE uri LIKE '%jpg%'
ORDER BY timetaken DESC limit 50;
Requests for JPG images sorted by download time, highest to lowest

Visualize CDN performance with Amazon QuickSight
• Pull data from Athena table created for CloudFront logs analysis
• Auto-detects data types to fit into graphs
CloudFront
log files

Visualize CDN performance with Amazon QuickSight
• Select visual type
• Line graphs for trends
over time
• Bar charts to compare
between locations
• Build a story
• Snapshot in time to
present to executives
• Share and export visuals
as needed

https://docs.aws.amazon.com/athena/latest/ug/cloudfront-logs.html

Operational analytics

The explosion of machine-generated data
Transition from IT
to DevOps
Increase in IoT and
Mobile Devices
Cloud-based
architectures
Machine-generated data is growing 10x faster than business data
Source: insideBigData - The Exponential Growth of Data, February 16, 2017

Operational analytics requirements/challenges

Challenges faced by customers – Log Analytics

Is there a better solution?
CloudWatch Logs
Log Management
Store and monitor your logs
Log Analytics
Get answers from your logs

Amazon
CloudWatch
Amazon CloudWatch is a
monitoring service for AWS
cloud resources, applications
you run on AWS and onPrem.
© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.22
Monitor EC2Spot Trends Monitor Other
Resources
Set Alarms -
Events
Monitor Custom
Metrics
Store & Analyze
Logs
Create
Dashboards
Take automated
action
Insights
Logs
Centralize
monitoring
Operational
Status
Troubleshoot

Complete Visibility of Your
Cloud Resources and
Applications
“Amazon CloudWatch monitors
more than 800 trillion metric
observations, triggers more than
2 trillion events, and ingests more
than 50 petabytes of logs per
month (*as of March 2018).”
Amazon
CloudWatch

Features & Benefits

Observability requires metrics, logs, and traces
CloudWatch Logs
Log Management
Store and Monitor your logs
CloudWatch
Logs
CloudWatch
Metrics
AWS X-Ray
Traces

CloudWatch Logs
CloudWatch Logs
Log Management
Store and Monitor your logs

CloudWatch Logs
CloudWatch
Logs
Comprehensive
Ingress
Amazon EC2
AWS Lambda
Amazon VPC Amazon ECS
Custom Logs
AWS CloudTrail
Flexible
Egress
Partner tools
Amazon Kinesis
Data Firehose
Amazon S3
Amazon
Elasticsearch Service
AWS Lambda
Integrates easily with AWS services and third-party solutions

CloudWatch Logs Insights
An interactive, fully integrated, and
pay-as-you-go log analytics service
for CloudWatch
Introducing

CloudWatch Logs Insights
CloudWatch

Simple Query Examples
Calculations
Filters
Counters
Summarizations
Percentiles
Timeseries aggregation
Sorting
Subsets
Full-text search
Column extraction from unstructured logs using the parse command
fields Operation, Bytes, Bytes/1024 as @KBs
filter Latency > 1000
stats count(), distinct(Operation) by Region
stats sum(Latency), avg(Latency), max(Latency) by
Operation
stats percentile(Latency, 99.9) by API
stats count() by bin(5m) as @myBin
sort @myBin desc
limit 20
Hello AND World
Task Example Query

Advanced Query - Examples
Filtering with calculated values
filter Operation = “MyOperation”
| fields @timestamp, Operation, Bytes/1000 as @KBs
Filtering, aggregate calculation by certain field, with sort
filter Operation != “SomeExcludedOperation”
| stats percentile(Latency, 99.9) as @My3ninesLatency by Operation
| sort @My3ninesLatency desc
| limit 10
Get a timeseries of some aggregates for log events that match a certain regex
filter Operation like /Kinesis/
| stats max(Latency) by bin(5m)

https://github.com/aws-samples/reinvent2018-dev303-code

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
The Amazon Elasticsearch Service

Amazon Elasticsearch Service is a cost-effective
managed service that makes it easy to deploy,
manage, and scale open-source Elasticsearch and
Kibana for log analytics, full-text search, and more.

Easy to Use
Deploy a production-ready Elasticsearch
cluster in minutes
Simplifies time-consuming management
tasks such as software patching, failure
recovery, backups, and monitoring
Open
Get direct access to the Elasticsearch open-
source API
Fully compatible with the open-source
Elasticsearch API, for all code and
applications
Secure
Secure Elasticsearch clusters with AWS
Identity and Access Management (IAM)
policies with fine-grained access control
access for users and endpoints
Automatically applies security patches
without disruption, keeping Elasticsearch
environments secure
Available
Provides high availability using Zone
Awareness, which replicates data between
two Availability Zones
Monitors the health of clusters and
automatically replaces failed nodes,
without service disruption
AWS Integrated
Integrates with Amazon Kinesis Firehose,
AWS IoT, and Amazon CloudWatch Logs for
seamless data ingestion
AWS CloudTrail for auditing, AWS Identity
and Access Management (IAM) for security,
and AWS CloudFormation for cloud
orchestration
Scalable
Scale clusters from a single node up to 20
nodes
Configure clusters to meet performance
requirements by selecting from a range of
instance types and storage options,
including SSD-powered EBS volumes
Amazon Elasticsearch Service Benefits

Application Monitoring & Root-cause Analysis
CASE STUDY: EXPEDIA
Logs, lots and lots of logs. How to cost effectively monitor logs?
Require centralized logging infrastructure
Did not have the man power to manage infrastructure
P R O B L E M
Quick insights: Able to identify and troubleshoot issues in real-time
Secure: Integrated w/ AWS IAM
Scalable: Cluster sizes are able to grow to accommodate additional log sources
B E N E F I T S
Streaming AWS CloudTrail logs, application logs, and Docker startup
logs to Elasticsearch
Created centralized logging service for all team members
Using Kibana for visualizations and for Elasticsearch queries
S O L U T I O N

Leading enterprises trust Amazon Elasticsearch Service for
their search and analytics applications
Media &
Entertainment
Online
Services
Technology Other

You can analyze field values to get statistics
and build visualizations
58,
12,
1,
100,
115
123,
214,
947
Result
GET
GET
POST
GET
PUT
GET
GET
POST
Field
Data
Buckets
GET
POST
PUT
5
2
1
Counts
Key Idea

ElasticSearch Service with Kibana

Get the right set up

Data is stored in indexes, distributed across shards
Amazon Elasticsearch Service domain
ID
Field: value
Field: value
Field: value
Field: value
Index
Shards
Instances
• Shards are primary or
replica
• Primary shard count
can’t be changed
• Elasticsearch distributes
shards to instances
elastically
• Primary and replica are
distributed to different
instances

How many instances?
• Index size is approximately source size
• Double this if you are deploying an index replica
• Instance count based on storage
requirements
• Either local storage or up to 1.5 TB of Amazon
Elastic Block Store (EBS) per instance
Example: a 2 TB corpus will need 4 instances
Assuming a replica and using EBS
Given 1.5 TB of storage per instance, this gives 6TB of storage

Instance type recommendations
Instance Workload
T2 Entry point. Dev and test. OK for dedicated masters.
M3, M4 Equal read and write volumes.
R3, R4 Read-heavy or workloads with high memory
demands (e.g., aggregations).
C4 High concurrency/indexing workloads
I2,I3 Up to 1.6 TB of SSD instance storage.

It is elastic
• Run out of storage space?
• add data nodes or
• increase EBS volume size.
• Need more compute or RAM?
• increase the instance type, or
• add more data nodes.
Amazon Elasticsearch Service is designed to apply those configuration
changes without incurring downtime.

Full text search
CASE STUDY: MirrorWeb
Make the UK Government and UK Parliament’s web archives searchable
Large scale ingestion scenario: 120 TB of data (1.2 MM 100MB files),
duplicates and bad data, Warc format
P R O B L E M
Scalability: Started on a 9-node, R4.4Xlarge cluster for fast ingest, reduced to 6 R4.Xlarge instances for search.
Able to reconfigure the cluster with no down time
Cost effective: Indexed 1.4 billion documents for $337
Fast: 146 MM docs per hour indexed. 14x faster than the previous best for this data set (using Hadoop)
B E N E F I T S
S O L U T I O N
Amazon
S3
(Storage)
Amazon
EC2
(Filtering)
Amazon
EC2
(Extraction)
Amazon ES
(Search)
For more on this case, see http://tinyurl.com/ybqwbolq

To scale correctly, iterate!
• Set config
• Instance type, storage
• Monitor via CloudWatch and
ES APIs
• CloudWatch for CPU, JVM,
and Disk
• ES for indexing, merges,
latencies
• Slow Logs for bottlenecks
• Adjust for workload
Configure
Cluster
Run your
workload
Gather
metrics

Use ES to analyze ES performance
https://aws.amazon.com/blogs/database/analyzing-amazon-elasticsearch-service-slow-logs-using-amazon-
cloudwatch-logs-streaming-and-kibana/

Data pattern for time series data (or logs)
Amazon ES cluster
logs_01.21.2017
logs_01.22.2017
logs_01.23.2017
logs_01.24.2017
logs_01.25.2017
logs_01.26.2017
logs_01.27.2017
Shard 1
Shard 2
Shard 3
host
ident
auth
timestamp
etc.
Each index has
multiple shards
Each shard contains
a set of documents
Each document contains
a set of fields and values
One index per day

Kinesis Firehose
CloudWatch Logs
Logstash
Amazon Lambda
Transform
Kinesis Firehose/
Streams
CloudWatch Logs
Amazon Elasticache/Redis
Kafka Rabbit MQLogstash
Amazon S3
Buffer
Kinesis Firehose/
Streams
Logstash
Worker nodes
Deliver
Kinesis Agent
CloudWatch Logs Agent
Beats
Fluentd
Application
Logstash
Collect
CloudWatch Logs
Amazon Lambda

Small-ish hosted use cases
Public Subnet
Internet Gateway
ALB
Private
Subnet
Filebeat
Redis ENI
Kibana
Amazon ES ENI Amazon ES Domain
Redis Cluster

Large or serverless use cases
• Data flows from instances and
applications via Lambda
• SigV4 signing via Lambda/roles

XL hosted use cases
• Ingest supported through high-
volume technologies like Spark or
Kinesis
• Up to 60 TB of data today

Demo. S3 to Lambda to Amazon
Elasticsearch for serverless ingestion
https://github.com/dbnegative/lambda-cloudfront-log-ingester

Streaming with Amazon Kinesis
Easily collect, process, and analyze video and data streams in real time
Capture, process,
and store video
streams
Load data streams
into AWS data
stores
Analyze data
streams in real time
Capture, process,
and store data
streams

Log processing at Netflix using Kinesis Data Streams
Netflix’s Amazon Kinesis Streams-based solution has proven to be highly scalable, each day
processing billions of traffic flows. Typically, about 1,000 Amazon Kinesis shards work in
parallel to process the data stream. “Amazon Kinesis Streams processes multiple terabytes of
log data each day, yet events show up in our analytics in seconds. We can discover and
respond to issues in real time, ensuring high availability and a great customer experience.”
“Our solution built on Amazon Kinesis enables us to identify ways to increase efficiency,
reduce costs, and improve resiliency for the best customer experience,”
John Bennett. Senior Software Engineer, Netflix

Unified logging
Log data in different
formats
Cross-service tracing almost
impossible
Forensics, monitoring, and
analytics on log data hard
SLO/SLI metrics from log data
impossible
Unified logging via
standardized log data
model
Annotate log records with
distributed tracing states
Based on open tracing
specification:
http://opentracing.io
SDK supporting multiple
languages

Sample UL log record
{
"ul-ctx-caller-span-id": "d62f04051cc5f312",
"ul-ctx-head-span-id": "1-5af39328-7cf81c9c9f3ac30af7693a65",
"ul-operation": "Operation_name",
"ul-span-duration": 160,
"ul-span-id": "d62f04051cc5f312",
"ul-span-start": 1525912360890,
"ul-tag-status": "INFO",
"ul-tag-ul-version": "2.1.0",
"ul-timestamp-epoch": 1525912361050
"ul-log-data": {
"cur_service_id": “xxxxxxxxxxxxxxxxxxx",
"cur_service_name": "xxxxxxx",
"method_name": "xxxx",
"module_name": "xxxxx",
"provider_name": "oxygen",
"scope": "C",
"sdk_version": "java_unified-logging-2.1.0",
"server_id": "xxxxxxx",
"significance": "INFO",
"src_service_id": "xxxxxx",
}
}

A unified log
Architecture
SQL
Processor
S P
O G
https://www.youtube.com/watch?v=jhYIxM2KiLs

Logs dashboard with Kibana

Performance and observability with x-ray

Metrics SPOG with Grafana

Thank you
Javier Ramirez
@supercoco9

Analyzing your web and application logs on AWS. Utrecht AWS Dev Day

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Analyzing your web and application logs on AWS. Utrecht AWS Dev Day

Similar to Analyzing your web and application logs on AWS. Utrecht AWS Dev Day (20)

More from javier ramirez

More from javier ramirez (20)

Recently uploaded

Recently uploaded (20)

Analyzing your web and application logs on AWS. Utrecht AWS Dev Day