As containers become more embedded in the platform tools, debug tools, traces and logs become increasingly important. Nare Hayrapetyan, Senior Software Engineer and Calvin French-Owen, Senior Technical Officer for Segment will discuss the principals of monitoring and debugging containers and the tools Segment has implemented and built for logging, alerting, metric collection, and debugging of containerized services running on Amazon ECS.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Monitoring, Logging and Debugging
Containerized Services
N a r e H a y r a p e t y a n , S e n i o r S o f t w a r e E n g i n e e r , A W S
C a l v i n F r e n c h - O w e n , C o - f o u n d e r , S e g m e n t
C O N 3 2 0
N o v e m b e r 2 8 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring, Logging, and Debugging
• Microservices are hard to monitor and debug
• A lot of interacting components:
• load balancers
• instances
• clusters
• tasks
• Containers are transient
• Need visibility into all parts

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring, Logging, and Debugging
• Can customers access my service?
• If not, what is failing?
• What trends and anomalies emerge?

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key Metrics
Infrastructure
• cpu, memory
• load balancers
• disk space

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key Metrics
Application
• error rates
• request volume
• request latencies

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring with Amazon CloudWatch
• ECS metrics are sent to Cloudatch
• Agent version >= 1.4.0
• CPU, memory reservation and utilization

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring with Amazon CloudWatch

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring with Amazon CloudWatch
• ECS provides 1 minute metrics
• High resolution metrics: 5 seconds
• Custom metrics: disk space

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring with Amazon CloudWatch

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring with Datadog

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring with Sysdig

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Logging
• Call volume is up!
• Error rates spiked!
• What’s going on?

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Logging
• Send logs to a logging framework
• ECS supports multiple log drivers
• Configure in task definition

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ECS Supported Log Drivers

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWSLogs Driver

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon CloudWatch Logs

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Alarms with Amazon CloudWatch
• How do you know something’s wrong?
• Page yourself!

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Alarms with Amazon CloudWatch

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Alarms with Amazon CloudWatch

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Debugging Task Failures
• New deployment
• Service is not launching tasks
• Tasks go into PENDING and disappear

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Debugging Service Failures

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Debugging Task Failures

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Debugging Task Failures

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Instance Logs
• Instance, ecs-agent and Docker daemon logs
in CloudWatch
• Install and configure CloudWatch logs agent

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Instance Logs

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Trends
• Billions of log lines
• Extract trends and patterns
• Search and analytics tools

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Trends
Amazon CloudWatch Logs
Amazon S3
Amazon Kinesis
AWS Lambda
Amazon Elasticsearch Service
Amazon ECS
Store
Stream
Process
Search

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Trends

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lessons learned
• Metrics and logs are critical
• Monitor at all levels
• Log and alarm
• Use tools to aggregate and visualize data and
pinpoint anomalies

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
SEGMENT
Calvin French-Owen Co-Founder

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
It’s 2 a.m., I’m getting paged…

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
It’s 2 a.m., I’m getting paged…
…now what?

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Segment by the numbers
- 140 billion monthly events
- 160k events/second peak
- 16,000 containers
- 350 Amazon Container
Service (ECS)

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Three steps to the (debugging) epiphany
1. Build your mental model
2. Dig in
3. Lean into the cloud

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1. Build your mental model

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Step 1: Build your mental model
- How does the system fit together?
- How is my service configured?
- What should I even be looking at?

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Step 1: Build your mental model
- How does the system fit together? specs
- How is my service configured?
- What should I even be looking at?

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
$ docker run segment/specs
https://github.com/segmentio/specs
Running specs

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Step 1: Build your mental model
- How does the system fit together? specs
- How is my service configured? specs and terraform
- What should I even be looking at?

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A terraform ECS service

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Terraform service—under the hood

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Step 1: Build your mental model
- How does the system fit together? specs
- How is my service configured? specs and terraform
- What should I even be looking at? Amazon CloudWatch and Datadog

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Step 1: Build your mental model
- How does the system fit together? specs
- How is my service configured? specs and terraform
- What should I even be looking at? Amazon CloudWatch and Datadog

52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2. Dig in

53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Step 2: Dig in
• Stats?
• Logging?
• Tracing?

54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Step 2: Dig in
• Stats? Amazon CloudWatch and statsd/veneur
• Logging?
• Tracing?

55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

57. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

58. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
service service service
veneur datadog-agent
api.datadog.com CloudWatch

59. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

60. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

61. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

62. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Step 2: Dig in
• Stats? Amazon CloudWatch and statsd/veneur
• Logging? CloudWatch, ECS-logs, cwlogs
• Tracing?

63. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CloudWatch Logs

64. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
…
RateLimitIntervalSec=1m
RateLimitBurst=200000
…
Journald.conf

65. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

66. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

67. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

68. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

69. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

70. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

71. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Logging
• github.com/segmentio/ecs-logs
• github.com/segmentio/ecs-logs-go
• github.com/segmentio/ecs-logs-js
• github.com/segmentio/rate-limiting-log-proxy
• github.com/segmentio/cwlogs

72. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Step 2: Dig in
• Stats? Amazon CloudWatch and statsd/veneur
• Logging? CloudWatch, ECS-logs, cwlogs
• Tracing? BCC, pprof-server

73. • BPF Compiler Collection
• No kernel modules, no
instrumentation
• A lot of very useful tools
• https://github.com/iovisor/bcc
BCC

74. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

75. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

76. • Pprof is automatically exposed
by the go runtime
• Gives you profiling, heatmaps,
memory dumps, and more
• Nice visualizations, one URL
click away
• Server exposed by consul
• github.com/segmentio/pprof-server
Pprof-server

77. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

78. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

79. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

80. $ docker run -it --rm -p 6061:6061
segment/pprof-server
-registry consul://172.17.0.1:8500
github.com/segmentio/pprof-server
Pprof-server

81. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Step 2: Dig in
• Stats? Amazon CloudWatch and statsd/veneur
• Logging? CloudWatch, ECS-logs, cwlogs
• Tracing? BCC, pprof-server

82. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
3. Lean into the cloud

83. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Step 3: Lean into the cloud
- Cattle, not pets
- Auto-scale and pre-scale

84. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cattle, not pets
- Reproduceable machine images
- Built with packer
- Run via systems
- Out-of-the-box autoscaling
- Created with terraform
- github.com/segmentio/roll-instances

85. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

86. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

87. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

88. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Auto Scaling everywhere
- Comes default with any service
- Tuned for systems like Amazon DynamoDB
- No autoscaling==not ready for prod

89. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

90. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lessons learned
Build tools to surface actionable information first
Auto-scaling a huge win
Give developers alerting + scaling policies out of the box
Passive tools are easy to build adoption around

91. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!
C L I C K T O A D D T E X T
C L I C K T O A D D T E X T