Successfully reported this slideshow.
Your SlideShare is downloading. ×

Debug your Container and Serverless Applications with AWS X-Ray in 5 Minutes - AWS Online Tech Talks

Check these out next

1 of 60 Ad
1 of 60 Ad

Debug your Container and Serverless Applications with AWS X-Ray in 5 Minutes - AWS Online Tech Talks

"Learning Objectives:
- Go beyond basic use cases and learn how to extract deeper value from X-Ray
- Leverage X-Ray to track and alert on performance trends and understand customer impact
- Learn best practices to monitor Serverless applications"

"Learning Objectives:
- Go beyond basic use cases and learn how to extract deeper value from X-Ray
- Leverage X-Ray to track and alert on performance trends and understand customer impact
- Learn best practices to monitor Serverless applications"

Advertisement
Advertisement

More Related Content

Slideshows for you (19)

Similar to Debug your Container and Serverless Applications with AWS X-Ray in 5 Minutes - AWS Online Tech Talks (20)

Advertisement

More from Amazon Web Services (20)

Advertisement

Debug your Container and Serverless Applications with AWS X-Ray in 5 Minutes - AWS Online Tech Talks

  1. 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Abhishek Singh, Sr. Manager Product Management (AWS) Ivonne Roberts, Software Architect (Financial Engines) Debug your Container and Serverless Applications with AWS X-Ray in 5 Minutes or less
  2. 2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What to Expect from the Session  What is X-Ray and how does it help?  Concepts  Demo - Debug an app in < 5 mins  How can you instrument your own app?  Customer Story  Q&A
  3. 3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What is X-Ray and how does it help?
  4. 4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Traditional Debugging Developer Local Test Developer Add Breakpoints Add Log Statements
  5. 5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Log-Centric Approach service 1 logs service 2 logs service 3 logs user
  6. 6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Challenges for microservices apps • Cross-service interactions • Varying log formats across services • Collecting, aggregating, and collating logs from services
  7. 7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Challenges for serverless apps • Resources typically only exist during execution. • Unlike traditional hosts, can’t install agents for monitoring. • Collecting, aggregating, and collating logs in real-time without latency overhead. • Events need to be correlated across services.
  8. 8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The traditional process of debugging doesn’t scale well for production applications or those built using a service- oriented, microservice, or serverless architecture. It’s tedious, repetitive, and time consuming.
  9. 9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. User-Centric Approach user Amazon DynamoDB Table Amazon SQS Queue Frontend API
  10. 10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How does AWS X-Ray help? • Analyze and debug performance of your distributed applications. • View latency distribution and pinpoint performance bottlenecks. • Identify specific user impact across your applications. • Works across different AWS and non-AWS services. • Ready to use in production with low latency in real-time. Enables you to get started quickly without having to manually instrument your application code to log metadata about requests
  11. 11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Visualize service call graph
  12. 12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Identify performance bottlenecks
  13. 13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Identify performance bottlenecks
  14. 14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Identify performance bottlenecks
  15. 15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Pinpoint issues
  16. 16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Identify errors
  17. 17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Identify errors
  18. 18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Identify impact to users
  19. 19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Lambda + X-Ray
  20. 20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Benefits of using AWS X-Ray with Lambda • X-Ray agent is natively built into Lambda. • Identify initialization and cold starts in Lambda • Pinpoint issues in downstream services called from your AWS Lambda function. • Happens with low latency in real-time. Can see traces in seconds.
  21. 21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Visualize Service Call Graph (Lambda)
  22. 22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Identify initialization & cold starts (Lambda)
  23. 23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Pinpoint errors (Lambda)
  24. 24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Additional Use Cases Use Off- Cloud Custom Payload Deep Linking Custom Apps via APIs Filter Expressions
  25. 25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. X-Ray Service
  26. 26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. App & X-Ray SDK EC2 Instance X-Ray Daemon Localhost UDP X-Ray API HTTPS HTTPS X-Ray Console App & X-Ray SDK Server X-Ray Daemon Localhost UDP EC2 Role AWS Credentials DevOps Team HTTPS X-Ray Workflow
  27. 27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. X-Ray Concepts
  28. 28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. X-Ray Concepts user Amazon DynamoDB Table Amazon SQS Queue Frontend API
  29. 29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. X-Ray Concepts Trace End-to-end data related a single request across services Segments Portions of the trace that correspond to a single service Sub-segments Remote call or local compute sections within a service Annotations Business data that can be used to filter traces Metadata Business data that can be added to the trace but not used for filtering traces Errors Normalized error message and stack trace Sampling Percentage of requests to your application to capture as traces
  30. 30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Let’s debug an app in <5 minutes!
  31. 31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How can you instrument your app?
  32. 32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. X-Ray SDK Enables you to get started quickly without having to manually instrument your application code to log metadata about requests Source on GitHub under https://www.github.com/aws/ Available for Java, .NET, .NET Core, Ruby, Python, Go, and Node.js. Adds filters to automatically captures metadata for calls to: • AWS services using the AWS SDK • Non-AWS services over HTTP and HTTPS (3rd party APIs) • Databases (MySQL, PostgreSQL, and Amazon DynamoDB) • Queues (Amazon SQS)
  33. 33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. App Instrumentation (Node.js) //Add aws-xray-sdk package to package.json var XRay = require('aws-xray-sdk'); var AWS = captureAWS(require('aws-sdk')); … XRay.config([XRay.plugins.EC2]); XRay.captureHTTPs(http); XRay.setDefaultName('myfrontend-dev'); … app.use(XRay.express.openSegment()); app.get('/', function(req, res) … app.get('/blog', function(req, res) … app.use(XRay.express.closeSegment());
  34. 34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Adding Business Data (Node.js) //Example showing how to add business data to traces app.post('/signup', function(req, res) { var item = { 'email': {'S': req.body.email}, 'name': {'S': req.body.name}, 'preview': {'S': req.body.previewAccess}, 'theme': {'S': req.body.theme} }; var seg = XRay.getSegment(); seg.addAnnotation('email', req.body.email); seg.addAnnotation('theme', req.body.theme); seg.addAnnotation('previewAccess', req.body.previewAccess); //Store item to DB //Send sign-up notification to user }
  35. 35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Sampling Configuration { "rules": { “payment": { "id": 1, "service_name": "*", "http_method": "*", "url_path": "/api/payment/*", "fixed_target": 0, "rate": 1.00 }, "base": { "id": 2, "service_name": "*", "http_method": "*", "url_path": "*", "fixed_target": 1, "rate": 0.1 } } } This example defines two rules: The first rule applies a 100% sampling rate with no minimum number of requests to trace to requests with paths under /api/payment The second overrides the default sampling rule with a rule that traces the first request each second and 10% of additional requests
  36. 36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. X-Ray Daemon Receives data from the SDK over UDP and acts as a local buffer. Data is flushed to the backend every second or when the local buffer fills. Available for Amazon Linux AMI, RHEL, Ubuntu, OS X, and Windows. Pre-installed on AWS Lambda. Can be run anywhere as long as AWS credentials are provided (for example, EC2, ECS, Kubernetes, on-premise, developer machine, and others).
  37. 37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. X-Ray API X-Ray provides a set of APIs to enable you to send, filter, and retrieve trace data You can send trace data directly to the service without having to use our SDKs (that is, you can write your own SDKs for languages not currently supported) Raw trace data is available using batch get APIs You can build your own data analysis applications on top of the data collected by X-Ray
  38. 38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. X-Ray API PutTraceSegments Uploads segment documents to AWS X-Ray BatchGetTraces Retrieves a list of traces specified by ID GetServiceGraph Retrieves a document that describes services in your application and their connections GetTraceSummaries Retrieves IDs and metadata for traces available for a specified time frame using an optional filter
  39. 39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How-to guides, blogs, and samples Lambda: o https://docs.aws.amazon.com/lambda/latest/dg/lambda-x-ray.html o https://aws.amazon.com/blogs/compute/analyzing-performance-for-amazon-rekognition-apps-written-on- aws-lambda-using-aws-x-ray/ o https://github.com/aws-samples/aws-xray-rekognition-lambda-sample o https://github.com/aws-samples/eb-java-scorekeep/tree/lambda Containers: o https://docs.aws.amazon.com/xray/latest/devguide/scorekeep-ecs.html o https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-ecs.html o https://aws.amazon.com/blogs/compute/application-tracing-on-kubernetes-with-aws-x-ray/ o https://github.com/aws-samples/aws-xray-kubernetes o https://github.com/aws-samples/aws-workshop-for-kubernetes
  40. 40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Trace ID & Propagation
  41. 41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Trace ID & Propagation
  42. 42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Trace ID & Propagation
  43. 43. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Trace ID & Propagation
  44. 44. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Trace ID & Propagation
  45. 45. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Trace ID & Propagation
  46. 46. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Trace ID & Propagation
  47. 47. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Trace ID & Propagation
  48. 48. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Trace ID & Propagation
  49. 49. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Customer Story
  50. 50. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Ivonne Roberts Software Architect, Office of CTO Financial Engines, Inc. Twitter: @ivlo11 LinkedIn: https://www.linkedin.com/in/ivonneroberts/ With roots in Silicon Valley, Financial Engines is the nation’s largest independent investment advisor1. We believe that all Americans—not just the wealthy—should have access to high-quality, unbiased financial help, and our clients’ best interests should always come first. We offer financial help to more than 9 million people across over 700 companies 1 For independence methodology and ranking, see InvestmentNews RIA Data Center. (http://data.investmentnews.com/ria/).
  51. 51. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Architecture • Migrating from a monolith codebase on premise • First with a lift-and-shift hybrid approach • And then breaking pieces off into microservices in a serverless architecture with a high bar for using non-managed services • 15 microservices in production in AWS
  52. 52. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring Journey Use Cases: • Measure performance improvement as we move to the cloud • Ongoing measurement of performance during updates and changes • Able to identify and resolve issues fast Monitoring Implementations: • Instrument code with timing calculations and writing that to the end of the log • Stress test applications in dev that scraped Lambda’s log using LogType.TAIL • CloudWatch and Splunk dashboards and alerts Monitoring Customer Experience: • AWS X-Ray instrumentation and monitoring
  53. 53. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS X-Ray + Lambda Quick Wins: • Able to instrument our applications with very little effort • Able to see a sampling of user traffic • Average transactions per minute • Average latency • Response distribution • Able to quickly find and address performance concerns • Increased granularity with annotations • Slice and dice based on type of computation, company, number of accounts, etc. Couple of Notes: • You won’t get all errors sampled unless you add: forceSamplingOfCurrentSegment() • Can’t annotate root segment so you need to add a “root” sub-segment with annotation and metadata • Inferred Segment based off of sub-segments with services that don’t issue their own segments
  54. 54. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo
  55. 55. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Highlights Able to instrument our applications with very little effort Latency across different resources in a microservice Slice and dice reports with annotations
  56. 56. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Additional Information
  57. 57. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Quick Recap • Serverless Apps have built in logging & monitoring capabilities • Easy to extend with CloudWatch Metrics, Amazon ES, and third- party tools • AWS X-Ray is powerful tool to visualize and troubleshoot issues
  58. 58. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. X-Ray Pricing Free tier o The first 100,000 traces recorded are free o The first 1,000,000 traces retrieved or scanned are free Additional charges o Beyond the free tier, traces recorded cost $5.00 per million traces o Beyond the free tier, traces retrieved or scanned cost $0.50 per million traces
  59. 59. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Helpful Links Go to https://aws.amazon.com/xray to learn more Documentation: http://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html Samples: o .NET: https://github.com/awslabs/aws-xray-dotnet-webapp o Java: https://github.com/awslabs/eb-java-scorekeep/tree/xray o Node.js: https://github.com/awslabs/eb-node-express-sample/tree/xray o Python: https://github.com/awslabs/eb-py-flask-signup/tree/xray o Lambda: https://github.com/awslabs/aws-xray-rekognition-lambda-sample o Alarms & Alerts: https://github.com/aws-samples/aws-xray-cloudwatch-event o Heatmap & Trends: https://github.com/aws-samples/aws-xray-scatter-sample
  60. 60. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Editor's Notes

  • Traditional debugging involves:
    A development environment
    Search through logs for clues to reproduce the issue
    Set breakpoints in code to stop execution, and inspect variables and call stack
    Add additional log statements as necessary and redeploy application
    Repeat until the issue is fixed
  • Speaker notes: With AWS X-Ray integration you will see the complete end-to-end overview of your distributed application.
  • Speaker notes: You can click on a specific node to see the high-latency requests to that service. In this case, the DynamoDB node is selected, and we are selecting high-latency requests from the latency histogram and clicking on view traces. We would like to see what is happening with these requests.
  • Speaker notes: In this case, DynamoDB is taking a long time. Usually DDB returns back within a few milliseconds, but this request has taken 2.1 seconds to get back. Let us click on the DDB in the trace timeline and see whats happening.
  • Speaker notes: It has taken multiple retries. Using X-Ray, you can quickly zero-in on the performance bottleneck and understand why it is happening. In this case, you can reach out to the DDB team with the requestID to see what is the actual reason on multiple retries.
  • Speaker notes: You can also quickly pinpoint an issue using AWS X-Ray. This exception stack is showing you had a conditional check failed exception at the specific line-item in your code.
  • Speaker notes: X-Ray also helps you identify errors in your services. Here you can see that I can quickly group my traces using the status code and see that 100% of my traces within a specific time frame had 409 error. The trace list is at the bottom to drill-down further into specific traces.
  • Speaker notes: In this case I have an exception/error from DynamoDB. I can quickly identify the root cause of the error using X-Ray.
  • Speaker notes: Finally, I can use annotations feature in X-Ray to quickly identify impact for specific user. Here a user with this specific email is having all his requests failed. We can quickly drill down and see whats happening in the traces.
  • Speaker notes: When you enable X-Ray for Lambda you get a complete overview of your Lambda application and other downstream services. You can notice that there is an error in Rekognition. X-Ray helps you drill-down and look at specific error happening in Rekognitiion.
  • EV: JH to fix.
    JH: agreed, integrate with previous themes
  • JH: we need to do a diagram of xray daemon on an instance, buffering messages, filtering/ dropping some, passing to x-ray API -> passing to a dashboard..

    Explain that Daemon listens locally on UDP

    Receives data from the SDK over UDP and acts as a local buffer. Data is flushed to the backend every second or when the local buffer fills.
    Available for Amazon Linux AMI, RHEL, Ubuntu, OS X, and Windows
  • Trace - End-to-end data related a single request across services

    Segments - Portions of the trace that correspond to a single service

    Sub-segments - Remote call or local compute sections within a service




×