CON320_Monitoring, Logging and Debugging Containerized Services

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS re:INVENT Monitoring, Logging and Debugging Containerized Services N a r e H a y r a p e t y a n , S e n i o r S o f t w a r e E n g i n e e r , A W S C a l v i n F r e n c h - O w e n , C o - f o u n d e r , S e g m e n t C O N 3 2 0 N o v e m b e r 2 8 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring, Logging, and Debugging • Microservices are hard to monitor and debug • A lot of interacting components: • load balancers • instances • clusters • tasks • Containers are transient • Need visibility into all parts

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring, Logging, and Debugging • Can customers access my service? • If not, what is failing? • What trends and anomalies emerge?

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring with Amazon CloudWatch • ECS metrics are sent to Cloudatch • Agent version >= 1.4.0 • CPU, memory reservation and utilization

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring with Amazon CloudWatch • ECS provides 1 minute metrics • High resolution metrics: 5 seconds • Custom metrics: disk space

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Trends Amazon CloudWatch Logs Amazon S3 Amazon Kinesis AWS Lambda Amazon Elasticsearch Service Amazon ECS Store Stream Process Search

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Lessons learned • Metrics and logs are critical • Monitor at all levels • Log and alarm • Use tools to aggregate and visualize data and pinpoint anomalies

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Segment by the numbers - 140 billion monthly events - 160k events/second peak - 16,000 containers - 350 Amazon Container Service (ECS)

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Step 1: Build your mental model - How does the system fit together? - How is my service configured? - What should I even be looking at?

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Step 1: Build your mental model - How does the system fit together? specs - How is my service configured? - What should I even be looking at?

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Step 1: Build your mental model - How does the system fit together? specs - How is my service configured? specs and terraform - What should I even be looking at?

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Step 1: Build your mental model - How does the system fit together? specs - How is my service configured? specs and terraform - What should I even be looking at? Amazon CloudWatch and Datadog

51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Step 1: Build your mental model - How does the system fit together? specs - How is my service configured? specs and terraform - What should I even be looking at? Amazon CloudWatch and Datadog

71. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Logging • github.com/segmentio/ecs-logs • github.com/segmentio/ecs-logs-go • github.com/segmentio/ecs-logs-js • github.com/segmentio/rate-limiting-log-proxy • github.com/segmentio/cwlogs

72. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Step 2: Dig in • Stats? Amazon CloudWatch and statsd/veneur • Logging? CloudWatch, ECS-logs, cwlogs • Tracing? BCC, pprof-server

73. • BPF Compiler Collection • No kernel modules, no instrumentation • A lot of very useful tools • https://github.com/iovisor/bcc BCC

76. • Pprof is automatically exposed by the go runtime • Gives you profiling, heatmaps, memory dumps, and more • Nice visualizations, one URL click away • Server exposed by consul • github.com/segmentio/pprof-server Pprof-server

80. $ docker run -it --rm -p 6061:6061 segment/pprof-server -registry consul://172.17.0.1:8500 github.com/segmentio/pprof-server Pprof-server

81. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Step 2: Dig in • Stats? Amazon CloudWatch and statsd/veneur • Logging? CloudWatch, ECS-logs, cwlogs • Tracing? BCC, pprof-server

84. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cattle, not pets - Reproduceable machine images - Built with packer - Run via systems - Out-of-the-box autoscaling - Created with terraform - github.com/segmentio/roll-instances

88. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Auto Scaling everywhere - Comes default with any service - Tuned for systems like Amazon DynamoDB - No autoscaling==not ready for prod

90. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Lessons learned Build tools to surface actionable information first Auto-scaling a huge win Give developers alerting + scaling policies out of the box Passive tools are easy to build adoption around

CON320_Monitoring, Logging and Debugging Containerized Services

CON320_Monitoring, Logging and Debugging Containerized Services

Recommended

More Related Content

What's hot

What's hot(20)

Similar to CON320_Monitoring, Logging and Debugging Containerized Services

Similar to CON320_Monitoring, Logging and Debugging Containerized Services(20)

More from Amazon Web Services

More from Amazon Web Services(20)

CON320_Monitoring, Logging and Debugging Containerized Services