Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to re:Invent 2019 CON328R Improving observability of your containers
Similar to re:Invent 2019 CON328R Improving observability of your containers (20)
Recently uploaded
Recently uploaded (20)
re:Invent 2019 CON328R Improving observability of your containers
- 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Improving observability of your containers C O N 3 2 8 - R Shubha Rao Principal Product Manager AWS Akshay Ram Sr. Product Manager AWS Sharanya Devaraj Software Development Engineer AWS
- 3. What does improving observability mean? Full stack visibility: Customers wanted to get visibility into the different layers of your stack, especially the app level
- 4. Visibility into all layers of the application stack Fullstackvisibility Application level: Each service, between services Container service level: Services, tasks, pods Compute service level: Clusters, instances
- 5. What does improving observability mean? Compatibility with existing tools: Customers wanted our container services to work well with current systems
- 6. Compatibility with existing toolset Compatibility with existing toolset AWS-Managed: Amazon CloudWatch, AWS X-Ray etc. APN Partner: Datadog, Splunk, Sysdig etc. Self-Managed: Prometheus, ELK stack, etc.
- 7. What does improving observability mean? Run anywhere: Customers wanted to have the same observability independent of their compute (Amazon EC2 or AWS Fargate) or orchestrator (Amazon ECS or Amazon EKS)
- 8. Flexibility to run anywhere without changing tools Deployment, scheduling, scaling, and management of containerized applications Where the containers run App 1 Amazon Elastic Kubernetes Service (Amazon EKS) Amazon EC2 AWS Fargate Orchestration Compute Engine Application Amazon Elastic Container Service (Amazon ECS) App 2 App 3 Built by different teams with different programming languages and protocols, and deployed on different services
- 9. Use the same tools across compute options Runonanycomputeoption Amazon ECS Amazon EKS Amazon EC2 mode AWS Fargate
- 10. Observability experiences that support all optionsFullstackvisibility Application Container service Compute Compatibility with existing toolset AWS-Managed APN Partner Self-Managed Runonanycomputeoption Amazon ECS Amazon EKS Amazon EC2 mode AWS Fargate
- 11. In this talk, you will hear more about…Fullstackvisibility Application Container service Compute Compatibility with existing toolset AWS-Managed APN Partner Self-Managed Runonanycomputeoption Amazon ECS Amazon EKS Amazon EC2 mode AWS Fargate AWS Outposts
- 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 13. AWS App Mesh: Managing application communications across AWS Consistency across teams Failure visibility and isolation Communication management Fine-grained deployment controls
- 14. My App What is AWS App Mesh? My App Search My App My App My App Recommendations Frontend Service mesh to manage communication between services
- 15. What is AWS App Mesh? Search Service mesh to manage communication between services Proxy SearchProxy SearchProxy RecommendationsProxy RecommendationsProxy RecommendationsProxy RecommendationsProxy Frontend Proxy Frontend Proxy Frontend Proxy Frontend Proxy Frontend Proxy
- 16. What is AWS App Mesh? Search Service mesh to manage communication between services Proxy SearchProxy SearchProxy RecommendationsProxy RecommendationsProxy RecommendationsProxy RecommendationsProxy Frontend Proxy Frontend Proxy Frontend Proxy Frontend Proxy Frontend Proxy
- 17. What is AWS App Mesh? Search Service mesh to manage communication between services Proxy SearchProxy SearchProxy RecommendationsProxy RecommendationsProxy RecommendationsProxy RecommendationsProxy Frontend Proxy Frontend Proxy Frontend Proxy Frontend Proxy Frontend Proxy Metrics, logs and tracing solution
- 18. What is AWS App Mesh? Search Service mesh to manage communication between services Proxy SearchProxy SearchProxy RecommendationsProxy RecommendationsProxy RecommendationsProxy RecommendationsProxy Frontend Proxy Frontend Proxy Frontend Proxy Frontend Proxy Frontend Proxy Amazon ECS Amazon EKS Amazon EC2 AWS App Mesh Kubernetes on AWS AWS Fargate
- 19. App Mesh works with Amazon CloudWatch Frontend Proxy CloudWatch Agent AWS-ManagedCustomer-Managed
- 20. App Mesh works with AWS Partner Network (APN) Partner solutions Frontend Proxy Statsd Agent APN Partner-ManagedCustomer-Managed
- 21. AWS App Mesh works with self-managed solutions Frontend Proxy /stats/prometheus Self-ManagedCustomer-Managed
- 22. App Mesh powers dashboards to troubleshoot Per service dashboard Service to service dashboard Tracing dashboards
- 23. CloudWatch Dashboards for front-end service
- 26. Get application-centric visualization using AWS X-Ray
- 27. Amazon CloudWatch ServiceLens brings logs, traces, and metrics together* NEW!!
- 28. AWS App Mesh enriches application-level visibilityFullstackvisibility Application Container service Compute Compatibility with existing toolset AWS-Managed APN Partner Self-Managed Runonanycomputeoption Amazon ECS Amazon EKS Amazon EC2 mode AWS Fargate AWS Outposts
- 29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 30. Compatibility with existing tools Challenges • Need support for a wider array of analytics and storage tools • Decouple configuration and lifecycle management of telemetry software from underlying compute Solution • Independent configuration and lifecycle of telemetry software • Extensive community as built on Fluent Bit and compatible with Fluentd • AWS manages the lifecycle of the Fluent Bit image
- 31. FireLens: One interface, many destinations One interface, many destinations Send logs natively to AWS and APN Partner log analytics and storage tools Levers to reduce costs You can send logs to cold storage and pull on demand to analytics tools De-couples log ingestion pipelines You can configure the log routing separately through a config file that is decoupled from the application
- 32. FireLens: Under the hood https://aws.amazon.com/blogs/containers/under-the-hood- firelens-for-amazon-ecs-tasks/
- 33. FireLens: Interface //App container "logConfiguration": { "logDriver":"awsfirelens", "options": { "Name": "datadog | sumologic | splunk | loggly | Kinesis Firehose | Kinesis Data Streams| CloudWatch", "apiKey": "<API_KEY>", } "secretOptions": [{ "name": "apiKey", "valueFrom": "arn:aws:secretsmanager:region:aws_account_id:secret:secret_name- AbCdEf" }] } //FluentBit sidecar { "essential":true, "image":"amazon/aws-for-fluent-bit:latest", "name":"log_router", "firelensConfiguration":{ "type":"fluentbit", "options":{ "enable-ecs-log- metadata":"true" } } }
- 34. FireLens enables sending logs to almost any destinationFullstackvisibility Application Container service Compute Compatibility with existing toolset AWS Managed Partner Self Managed Runonanycomputeoption ECS EKS EC2 mode AWS Fargate
- 35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 36. Container Platform Agnostic Challenges • Customers had teams that self preferred differing container services products • Mix of using self-managed and fully managed services for monitoring • Consistent interface to observe services running across environments Solution • Launched Container Insights, which works across ECS, EKS and AWS Fargate
- 37. CloudWatch Container Insights Agent Dashboards Events Logs Metrics A fully managed observability service for monitoring, troubleshooting and alarming on your containerized applications and microservices. Reliable, secure metrics and logs collection Automated summaries and analysis Observability experience across metrics, logs, traces Pre-created dashboards Alarms
- 38. CloudWatch Container Insights – Performance Logs Performance log events from ECS, EKS, Fargate CloudWatch Aggregation at various levels depending on service CloudWatch Dashboards with Metrics and Performance Logs
- 39. Enabling Container Insights on ECS/Fargate
- 40. Enabling Container Insights on EKS
- 41. Container Insights provides dashboards for troubleshootingFullstackvisibility Application Container service Compute Compatibility with existing toolset AWS Managed Partner Self Managed Runonanycomputeoption ECS EKS EC2 mode AWS Fargate
- 42. Use Cases • Debugging at the infrastructure level • Clusters, Nodes/Instances resource utilization, health information • Example: Cluster CPU, Memory consumption • Debugging at the Container Service level • Services, Tasks/Pods, Container level metrics and logs • Example: Service task count, Task/Pod Memory Utilization • Debugging at the Application level • CloudWatch Application logs • App Mesh - Envoy Metrics • Example – Log insights querying for application specific information
- 43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 44. AWS Cloud
- 45. Thank you! © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Shubha Rao shubha@amazon.com Akshay Ram akshram@amazon.com Sharanya Devaraj sharanyd@amazon.com
- 46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 52. Backend Services
- 53. Application Load
- 55. AWS Cloud
- 60. AWS Cloud
- 63. Add more fields to Dashboard
- 66. Envoy Metrics from App Mesh
- 73. Performance Logs
- 74. Log Insights
- 75. More Performance Logs Querying
- 76. Application Logs
- 77. CloudWatch Logs
- 80. App Mesh Request Metrics
- 81. Thank you! © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Shubha Rao shubha@amazon.com Akshay Ram akshram@amazon.com Sharanya Devaraj sharanyd@amazon.com
- 82. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Editor's Notes
- Talk about how customers need observability here… while there exists a lot of instrumentation on the instance and the service level, applications was a bling spot. Every team uses a differnt languages , every team uses a different layer some serverless some manage clusters so clusters are important
- The current AWS container services landscape covers a broad set of products. At the orchestration layer we’ve Amazon ECS and Amazon EKS. EKS makes it easy to deploy, manage, and scale containerized applications using Kubernetes on AWS. You can currently run your containers on ECS using either the EC2 launch type – where get to manage the the underlying instances on which your containers are running - or you can choose to run your containers in a serverless manner with the AWS Fargate launch type. Finally, we provide a registry services, Amazon ECR, where you can store your container images.
- Another recent release that we recently shipped working from customer use cases is Firelens. Firelens was build working from feedback from customers, especially on AWS Fargate. For context when Fargate shipped at reinvent in 2017 we had only cloudwatch support. Customers who fall in to the patner, other AWS or self managed segmented wanted a better way to send their log data from containers running on fargate to their preffered tool of choice.
- Another recent release that we recently shipped working from customer use cases is Firelens. Firelens was build working from feedback from customers, especially on AWS Fargate. For context when Fargate shipped at reinvent in 2017 we had only cloudwatch support. Customers who fall in to the partner or other AWS or self managed segmented wanted a better way to send their log data from containers running on fargate to their preferred tool of choice. As we spoke to customers further we realized here are different teams that manage ingestion pipelie.
- We contributed Kinesis Firehose, Kinesis Data Streams, Cloud watch fluent bit plugins. Customers can use these to get to S3, Amazon Elasticsearch, Redshit We worked with partners to contribute to or vet their Fluentd solutions You can use FluentBit features to send logs to outputs based on importance. Lets say you want to send logs straight to S3 you can keep logs there and pull later on demand to tools We noticed sometime there are two separate teams that configure the log ingestion from the application. Independently configure ingestin
- This is how it works under the hood. We build all plubing from container to a sidecar Fluent Bit container. They talk via a unix socket.
- Clearly you can see there are a lot of options to send to any destination. What this means is customers get compatability which ever service they use, even a fully managed service like AWS Fargate. Now let me hand it to Sharanya who is going to speak about Container Insights and how it helps in the context of better observability.
- Initially, we learned from customers that there were teams in the organization that preferred using different container orchestration solutions for building applications, be it microservices/ devops. This was mainly due to different developer experiences and ease of use. With different container services, they used different tools for getting observability into their applications. For eg: in ECS/Fargate world, customers relied on cloudwatch for getting cluster/service level monitoring. EKS customers integrated 3rd party solutions such as Prometheus to get observability into their clusters. Many times, these different microservices talked to each other in some way or the other. Hence the missing piece here was a single unified tool that provides observability for services running across different platforms.
- Lets dive into what container insights is. Container insights is a fully managed observability service integrated with amazon cloud watch for providing visibility into your containerized applications and micro services What does container insights provide? Metrics aggregated on different levels of the stack namely infrastructure, application, ECS/EKS clusters that contains EC2 instances, you could get cluster/instance or node cpu,memory, disk utilization. It drills down even further and provides metrics and logs at the task or pod level and even down to individual containers. What could you do with it? Cloudwatch provides mechanisms for monitoring, alarming which customers could leverage for debugging when say there is an operation issue. It also provides automated dashboards at different aggregation level for better visualization of the metrics
- How does container insights do that ? Cloud watch gets it metrics from the respective container orchestrators in the form of performance log events in a JSON schema that contain the metrics. Cloudwatch uses these log events to perform aggregations based on the container services for example: your ECS or fargate tasks or pods in EKS One good thing is these performance log events are available to customers as well, which when combined with log insights can be used for powerful querying and analysis of the applications.
- Now that we know what container insights is and how does it work., lets look at enabling container insights for ECS/fargate You could use the put account setting API for enabling container insights for all the new clusters that would be created using that account If you already have existing clusters, you could use update cluster level settings api to turn on container insights monitoring. or you could simply use the account settings field when you’re creating a new cluster if you want to turn on insights for selected clusters. A recent new feature with container insights is you could get container instance level monitoring by deploying the cloudwatch agent as a daemon service
- - for EKS, its simple. you deploy CW agent as a daemon set in your cluster. And there is a simple one click setup script for that available in the container insights documentation.
- Let us go over the architecture of the application. There is a frontend service that is a ruby on rails application running on top of fargate, since it’s a serverless workload. It talks to a couple of backend services. One is a crystal backend service that’s again running on top of fargate and the other backend service is a nodejs application running on top of EKS. Each service has about 3 copies of the task running.