Cloud is the new normal, and organizations are deploying different types workloads on AWS. Understanding the performance efficiency and overall application performance is critical to ensuring that you can scale your workload to meet the demands of your customers. Understanding how well your application performs over time helps you to continuously improve and innovate your software to get the most out of the AWS platform. If you aren't measuring custom application metrics, you are operating your software blindly and cannot pinpoint areas of improvement. Learn how to use Amazon CloudWatch custom metrics, alerts, dashboards and AWS X-Ray to architect an application monitoring service to provide insight to your workload's performance.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application Performance
Management (APM) on AWS
Mar c os Or tiz , Solutions Ar c hitec t
N ovember 2017
A R C 3 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What to expect from this session
Look at industry
trends impacting
monitoring
How to use Amazon CloudWatch
custom metrics, logs, and AWS
X-Ray to get custom insights
about your application.
Understand some APM
on AWS best practices
See some of those
scenarios in action

3. Why APM is important
Every application is unique
Libraries
Programming
language
External
Dependencies
Performance
Requirements
Frameworks
Architecture
Synchronous
Asynchronous
Serverless
Containers
Monolithic
Protocols

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Do you really know your application?
Infrastructure
impacts
Bottlenecks
When/how to
scale
New versions
Traffic
volumes
Error rates
How can you manage what you don’t know?

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Trends in monitoring
Applications are more dynamic
Complexity is increasing
Increasing business impact

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How to perform APM on AWS
Custom
metrics
Logs
Measurement of
an event
Human-readable
events Amazon CloudWatch Logs
Amazon CloudWatch
Examples
Throughput, error
count, request rate,
request calculations,
queue sizes
Process output,
event correlation,
root cause analysis
AWS service
AWS X-Ray
Traces
Cross-service
view of requests
Review request
behavior, improve
app performance,
service mapping

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The challenge
PutMetricData is a
synchronous HTTP
request. It introduces
latency in your code.
The solution
Use an UDP agent like
collectd or logstash.* **
Amazon CloudWatch custom metrics
*CloudWatch plugin for collectd: https://aws.amazon.com/blogs/aws/new-cloudwatch-plugin-for-collectd/
**CloudWatch output for logstash: https://github.com/logstash-plugins/logstash-output-cloudwatch

8. Custom metrics architecture
Your app
EC2 instance
UDP
daemon
localhost
UDP
CloudWatch
API
HTTPS
HTTPS
AWS Management Console
EC2 role
DevOps team
HTTPS
Your app
UDP
daemon
localhost
UDP
AWS
credentials
Server
Fire and forget
Decouples metrics
aggregation and
API calls

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The challenge
1. Log centralization
2. Log archive
3. Source code access
and instrumentation
The solution
Use CloudWatch Logs agent.*
CloudWatch custom metrics from logs
*CloudWatch Logs agent reference: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html

10. CloudWatch Logs architecture
Your app
EC2 instance
CloudWatch
Logs agent
logs
CloudWatch
API
HTTPS
HTTPS
AWS Management Console
EC2 role
DevOps team
HTTPS
Your app
CloudWatch
Logs agent
AWS
credentials
Server
No need to
change your code
logs

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
From CloudWatch Logs to custom metrics
CloudWatch
Logs
Metric filters Custom metric
Works with space-delimited or
JSON log formats.

12. 1
2
3

13. Filter pattern – Space delimited logs
2017-09-08T22:33:06.877762 INFO 30223 0.001940809
2017-09-08T22:33:07.882269 INFO 73123 0.004267585
2017-09-08T22:33:08.882543 ERROR Random Error
2017-09-08T22:33:09.882809 ERROR Random Error
2017-09-08T22:33:10.887246 INFO 72689 0.004165085
2017-09-08T22:33:11.893134 INFO 98254 0.005619234
2017-09-08T22:33:12.896820 INFO 59771 0.003355367
[recorded_at, severity=INFO, item_count, elapsed_time]

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Filters pattern – Space delimited logs
2017-09-08T22:33:06.877762 INFO 30223 0.001940809
2017-09-08T22:33:07.882269 INFO 73123 0.004267585
2017-09-08T22:33:08.882543 ERROR Random Error
2017-09-08T22:33:09.882809 ERROR Random Error
2017-09-08T22:33:10.887246 INFO 72689 0.004165085
2017-09-08T22:33:11.893134 INFO 98254 0.005619234
2017-09-08T22:33:12.896820 INFO 59771 0.003355367
[recorded_at, severity=ERROR, msg]

15. Filter pattern – JSON Logs
{"recorded_at":"2017-09-08T22:33:06","severity":"INFO","key1":30223, "key2”:809}
{"recorded_at":"2017-09-08T22:33:07","severity":"INFO","key1":73123, "key2":585}
{"recorded_at":"2017-09-08T22:33:08","severity":"ERROR",”msg":"Random Error"}
{"recorded_at":"2017-09-08T22:33:09","severity":"ERROR",”msg":"Random Error"}
{"recorded_at":"2017-09-08T22:33:10","severity":"INFO","key1":72689, "key2":85}
{"recorded_at":"2017-09-08T22:33:11","severity":"INFO","key1":98254, "key2":234}
{"recorded_at":"2017-09-08T22:33:12","severity":"INFO","key1":59771,"key2":367}
{ $.severity = "INFO" }

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudWatch Logs subscriptions
CloudWatch
Logs
Lambda
function
Amazon ESSubscription Implement complex
parsing logic
Interact with other AWS
services
Anything you can code

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudWatch dashboards
Monitor multiple
resources in a
single view
Even those
resources that
spread across
multiple regions

18. Logging and UDP latency
EC2: t2.micro; EBS: 8GB, GP2
EC2: m4.large; EBS: 8GB, GP2
https://github.com/awslabs/cloudwatch-xray-apm-demo/blob/master/samples/latency/latency.rb

19. AWS X-Ray components
User
Amazon API
gateway
AWS Lambda
Amazon
DynamoDB
Amazon
SQS queue

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS X-Ray graph

21. AWS X-Ray diagram
Your app
EC2 instance
X-Ray
daemon
localhost
UDP
X-Ray
API
HTTPS
HTTPS
AWS Management Console
EC2 role
DevOps team
HTTPS
Your app
X-Ray
daemon
localhost
UDP
AWS
credentials
Server
Fire and forget
Decouples trace
aggregation and
API calls

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
//demo: putting-it-all-together
MyApp
collectd
Logs
X-Ray daemon
CloudWatch
Logs agent
CloudWatch
API
X-Ray
API
UDP
UDP
HTTPS
HTTPS
EC2 Instance
HTTPS
while true
factorial(n)
fibonacci(n)
log_metrics
udp_send_metrics
udp_send_xray_traces
sleep 1
end

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo
https://github.com/awslabs/cloudwatch-xray-apm-demo

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
APM on AWS best practices
•Design your apps to report custom metrics.
•Use an UDP daemon to avoid adding latency.
•Plan carefully your logging and metric strategy.
•Use X-Ray for distributed tracing and
troubleshooting.

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
APM on AWS best practices
•Deliver CloudWatch dashboards, alarms, and
self-healing automation as part of your app.
•Be aware of service limits such as
PutLogsEvents, PutMetricData,
PutTraceSegment, PutTelemetryRecords.
•Leverage AWS service’s metrics together with
your own custom metrics.

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Remember
Everything fails
all the time.
Werner Vogels, Amazon CTO

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Resources
https://aws.amazon.com/answers/logging/centralized-logging/
https://github.com/awslabs/cloudwatch-logs-centralize-logs
https://github.com/awslabs/cloudwatch-logs-customize-alarms
https://github.com/awslabs/cloudwatch-logs-analyze-data
https://github.com/awslabs/cloudwatch-dashboards-cloudformation-sample

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Complete your evaluations
Remember to complete
your evaluations in the
mobile app!

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
M a r c o s O r t i z
`