This document discusses distributed tracing in AWS using AWS X-Ray. It begins with an introduction to transaction monitoring concepts and challenges debugging distributed applications. It then provides an overview of AWS X-Ray for analyzing and debugging production applications, including its components, SDK instrumentation, and daemon. Finally, it demonstrates X-Ray features like trace filtering and advanced use cases before concluding.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jan Haak
Solutions Architect
Amazon Web Services
Level 300
Distributed Traceability in AWS
Evgeny Vaganov
Solutions Architect
Amazon Web Services
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Life of a Transaction

2. In the next 30 minutes

3. In the Next 30 Minutes
• Modern application & DevOps challenges
• Application monitoring concepts
• Tools
• A Demo of course
• Conclusion

4. This is a Level 300 Session
Enjoy the Ride

5. Transaction
A transaction symbolises a unit of work performed within an application.
Example: Request / Response pair.
> GET / HTTP/1.1
> Host: www.amazon.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/html;charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Server: Server
< Date: Wed, 15 Mar 2017 04:22:57 GMT
< Set-Cookie: skin=noskin; path=/; domain=.amazon.com
< Strict-Transport-Security: max-age=47474747;
< Cache-Control: no-cache
< X-Cache: Miss from cloudfront
< X-Amz-Cf-Id: 5lz_s7a1K06-Q6gFfeoR1nMH1RnMXy…
<

6. Log Centric
service 1 logs
service 2 logs
service 3 logs
user

7. User Centric
user
Amazon API
Gateway
AWS Lambda
Amazon
DynamoDB
Amazon
SQS queue

8. Demo

9. Challenges
Deploying and managing service-oriented applications is
more work compared to monolithic applications

10. Challenges
Services such as AWS Lambda, Amazon EC2 Container
Service, AWS Elastic Beanstalk, AWS CloudFormation,
etc. make it easier to deploy and manage applications
consisting of hundreds of services.

11. Challenges
Still hard to debug application issues in production applications due to:
• Cross-service interactions
• Varying log formats across services
• Collecting, aggregating, and collating logs from services

12. Debugging
Traditional debugging involves:
• A development environment
• Search through logs for clues to reproduce the issue
• Set breakpoints in code to stop execution, and inspect variables and call stack
• Add additional log statements as necessary and redeploy application
• Repeat until the issue is fixed
Traditional process of debugging doesn’t scale well to production
applications or those built using a service-oriented, microservice, or
serverless architecture
It’s tedious, repetitive, and time consuming

13. Right Tools for the Right Problem

14. AWS X-Ray (preview)
Analyse and debug production, distributed applications
Simple setup: instrument your application with X-Ray SDK
and install X-Ray Daemon
End-to-End Tracing, cross-service view of requests made to
your application.

15. AWS X-Ray (preview)
Service Map provides a view of connections between services
in your application and aggregated data for each service,
including average latency and failure rates.
Data Annotation and Filtering

16. AWS Service and Database Integrations
Amazon
ECS
Amazon
EC2
AWS
Elastic
Beanstalk
Amazon
SQS
Amazon
SNS
Amazon
RDS: Postgres,
MySQL, Aurora
Amazon
API
Gateway
Amazon
DynamoDB
AWS
Lambda

17. Concepts
Service Graph SamplingTracing Header Filter ExpressionsTrace ID

18. Trace List

19. Trace

20. AWS X-Ray Components
user
Amazon API
Gateway
AWS Lambda
Amazon
DynamoDB
Amazon
SQS queue

21. X-Ray Graph

22. Advanced Use Cases
Use off-cloud Custom
Payload
Deep Linking Custom Apps
via API
Filter
Expressions

23. The X-Ray API
API provides access to all functionality through
the AWS SDK.
AWS X-Ray is integrated with AWS Identity and Access
Management (IAM) so that you can control which users and
resources have permission to access your traces and how.

24. IAM Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"xray:PutTraceSegments",
"xray:PutTelemetryRecords"
],
"Resource": [
"*"
]
}
]
}

25. Instrument Your Application

26. Supported Programming Languages

27. The AWS X-Ray SDK for Node.js
1. Add the SDK to your application's dependencies.
*package.json*
"aws-xray-sdk" : "1.0.4-beta”
2. Initialise the SDK client and add use it in your application prior to declaring routes.
var app = express();
var AWSXRay = require('aws-xray-sdk');
app.use(AWSXRay.express.openSegment());
3. Use the SDK exceptions after declaring routes.
app.get('/', function (req, res) { res.render('index'); });
app.use(AWSXRay.express.closeSegment());

28. The AWS X-Ray SDK for Java
1. Add the SDK as a dependency in your build configuration.
*pom.xml*
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-xray-recorder-sdk-core</artifactId>
<version>1.0.3-beta</version>
</dependency>

29. The AWS X-Ray SDK for Java
2. Add a servlet filter to your deployment descriptor to trace incoming
HTTP requests.
*web.xml*
<filter>
<filter-name>XRayServletFilter</filter-name>
<filter-class>com.amazonaws.xray.javax.servlet.AWSXRayServletFilter
</filter-class> </filter>
<filter-mapping>
<filter-name>XRayServletFilter</filter-name>
<url-pattern>*</url-pattern>
</filter-mapping>

30. The AWS X-Ray SDK for .NET
1. Add the AWSXRayRecorder package to your application with NuGet.
2. Add the message handler to your HTTP configuration.
using Amazon.XRay.Recorder.Handler.Http;
public static class WebApiConfig
{
public static void Register(HttpConfiguration config)
{
config.MessageHandlers.Add(new TracingMessageHandler());
}
}

31. AWS X-Ray Daemon

32. App & X-Ray
SDK
EC2 Instance
X-Ray
Daemon
Localhost
UDP
X-Ray API
HTTPS
HTTPS
X-Ray Console
App & X-Ray
SDK
Server
X-Ray
Daemon
Localhost
UDP
EC2 Role AWS
Credentials
DevOps Team
HTTPS
AWS X-Ray Diagram

33. Daemon Installation (EC2 Windows Server)
EC2 User Data example for Windows:
<powershell>
if ( Get-Service "AWSXRayDaemon" -ErrorAction SilentlyContinue ) {
sc.exe stop AWSXRayDaemon
sc.exe delete AWSXRayDaemon
}
$targetLocation = "C:Program FilesAmazonXRay" if ((Test-Path $targetLocation) -eq 0) {
mkdir $targetLocation
}

34. Continues...
$zipFileName = "aws-xray-daemon-windows-service-1.x.zip”
$zipPath = "$targetLocation$zipFileName"
$destPath = "$targetLocationaws-xray-daemon"
if ((Test-Path $destPath) -eq 1) {
Remove-Item -Recurse -Force $destPath }
$daemonPath = "$destPathxray.exe"
$daemonLogPath = "$targetLocationxray-daemon.log"
$url = "https://s3.dualstack.ap-southeast-2.amazonaws.com/aws-xray-assets.ap-southeast-
2/xray-daemon/aws-xray-daemon-windows-service-1.x.zip"

35. Continues...
Invoke-WebRequest -Uri $url -OutFile $zipPath
Add-Type -Assembly "System.IO.Compression.Filesystem"
[io.compression.zipfile]::ExtractToDirectory($zipPath, $destPath)
New-Service -Name "AWSXRayDaemon" -StartupType Automatic -
BinaryPathName "`"$daemonPath`" -f `"$daemonLogPath`""
sc.exe start AWSXRayDaemon
</powershell>

36. Daemon Installation (EC2 Linux)
EC2 User Data installation script for Amazon Linux
#!/bin/bash
curl https://s3.dualstack.ap-southeast-2.amazonaws.com/aws-xray-assets.ap-
southeast-2/xray-daemon/aws-xray-daemon-1.x.rpm -o /home/ec2-
user/xray.rpm
yum install -y /home/ec2-user/xray.rpm

37. Daemon Installation (Amazon ECS)
To create a Docker image that runs the daemon:
1. Create a folder and download the daemon.
~$ mkdir xray-daemon && cd xray-daemon
~/xray-daemon$ curl https://s3.dualstack.ap-southeast-2.amazonaws.com/aws-
xray-assets.ap-southeast-2/xray-daemon/aws-xray-daemon-linux-1.x.zip -o ./aws-
xray-daemon-linux-1.x.zip
~/xray-daemon$ unzip -o aws-xray-daemon-linux-1.x.zip -d .

38. Continues...
2. Create a Dockerfile with the following content.
*~/xray-daemon/Dockerfile* FROM ubuntu:12.04 COPY xray /usr/bin/xray-daemon CMD xray-
daemon -f /var/log/xray-daemon.log &
3. Build the image.
~/xray-daemon$ docker build -t xray .

39. Daemon Installation (Elastic Beanstalk)
Elastic Beanstalk platforms released since December 2016 include the X-Ray
daemon. Enable the daemon with a configuration file or by setting the
corresponding option in Elastic Beanstalk console.
# .ebextensions/xray-daemon.config
option_settings:
aws:elasticbeanstalk:xray:
XRayEnabled: true

40. Demo

41. AWS CloudTrail
Track IAM principals activity and API usage

42. AWS Config
AWS Config provides AWS resource inventory, configuration history, and
configuration change notifications to enable security and governance.

43. Amazon CloudWatch and CloudWatch Logs
You can use Amazon CloudWatch to collect and track metrics, collect
and monitor log files, set alarms, and automatically react to changes in
your AWS resources.
Lambda Micro services
Amazon
CloudWatch
EC2 Application
Elastic Beanstalk service
Alarms
Alarm state:
“ALARM”
Debugging function
…..
…..
…..
Amazon SNS
Error queue Consumers
Ops team
Amazon
CloudWatch Logs

44. Amazon ElasticSearch Service
Amazon Elasticsearch Service makes it easy to deploy, operate, and
scale Elasticsearch for log analytics, full text search, application
monitoring, and more.
Amazon
ElasticSearch
Lambda Micro services
EC2 Application
Elastic Beanstalk service
Amazon
CloudWatch Logs
Ops team
Kibana Dashboard
Ad-hoc
Queries

45. AWS Partners

46. AWS Partners

47. Summary

48. Summary
• Complex distributed apps is the new normal.
• Debugging distributed apps is hard.
• AWS and AWS Partners have the right tools for the job.

49. Thank you!