Troubleshooting RabbitMQ and Microservices That Use It
•3 likes•1,010 views
Download to read offline
Report
SpringOne Platform 2016 Speaker: Michael Klishin; Software Engineer, Pivotal. Designing a system in terms of [micro] services is hype du jour but it's not without trade-offs. Debugging a distributed system can be challenging. In this talk we will cover how one can troubleshoot a distributed service-oriented system, focusing on Java, Spring, and RabbitMQ. Key points of the talk: -Multi-service system stability depends on how applications are designed as much as middleware -Multi-service systems are harder to debug -Several known tools and techniques exist to help: the audience will learn about them -Some problems are well understood and have known solutions: we will cover a few -Some tools in this spaces are underdeveloped or lacking "Structured platforms" help with troubleshooting and visibility, and affect how applications can/should be developed
Recommended
More Related Content
What's hot
What's hot(20)
Similar to Troubleshooting RabbitMQ and Microservices That Use It
Follow the White Rabbit - Message Queues with PHPEric Rodriguez (Hiring in Lex)
13.7K views•43 slides
Similar to Troubleshooting RabbitMQ and Microservices That Use It(20)
More from VMware Tanzu
More from VMware Tanzu(20)
Recently uploaded
Recently uploaded(20)
Troubleshooting RabbitMQ and Microservices That Use It
- 1. Troubleshooting RabbitMQ and services that use it
- 2. Who am I? • Staff Engineer, RabbitMQ @ Pivotal
- 3. Who am I? • Staff Engineer, RabbitMQ @ Pivotal • @michaelklishin, github.com/michaelklishin
- 5. The monolith problem
- 21. Troubleshooting publishers
- 22. Troubleshooting publishers • I/O exceptions (shutdown handlers)
- 23. Troubleshooting publishers • I/O exceptions (shutdown handlers) • Publisher confirms
- 25. When in doubt, borrow ideas from TCP
- 26. Troubleshooting publishers • I/O exceptions (shutdown handlers) • Publisher confirms • Returned message handlers
- 27. Troubleshooting publishers • I/O exceptions (shutdown handlers) • Publisher confirms • Returned message handlers • Invalid payload (e.g. fails to deserialize or decrypt)
- 28. Troubleshooting publishers • I/O exceptions (shutdown handlers) • Publisher confirms • Returned message handlers • Invalid payload (e.g. fails to deserialize or decrypt) • Identifying publisher instances
- 29. Troubleshooting publishers • identifying blocked (throttled) publishers
- 30. Client-provided connection names in RabbitMQ 3.6.3+
- 31. Troubleshooting publishers • identifying blocked (throttled) publishers • retries
- 32. Troubleshooting publishers • spring-amqp can cover all of the above
- 35. Troubleshooting consumers
- 36. Troubleshooting consumers • I/O exceptions
- 37. Troubleshooting consumers • I/O exceptions • Inadequate delivery QoS
- 38. Troubleshooting consumers • I/O exceptions • Inadequate delivery QoS • Lack of confirmations; double-confirming
- 40. Troubleshooting consumers • I/O exceptions • Inadequate delivery QoS • Lack of confirmations; double-confirming
- 41. Troubleshooting consumers • I/O exceptions • Inadequate delivery QoS • Lack of confirmations; double-confirming • Redelivery metrics
- 42. Troubleshooting consumers • I/O exceptions • Inadequate delivery QoS • Lack of confirmations; double-confirming • Redelivery metrics • Identifying consumer instances
- 43. Troubleshooting consumers • Consumer utilization (reported by HTTP API)
- 44. Troubleshooting consumers • spring-amqp can help with some of the above
- 46. — W. Edwards Deming “In God we trust, all others must bring data…”
- 47. — W. Edwards Deming “In God we trust, all others must bring data…”
- 48. — What do you do for a living?
- 49. — What do you do for a living? — Tell people to read the logs.
- 50. Sources of data useful for debugging
- 51. Sources of data useful for debugging • Metrics
- 52. Sources of data useful for debugging • Metrics • Your logs
- 53. Sources of data useful for debugging • Metrics • Your logs • Someone else's logs
- 54. Sources of data useful for debugging • Metrics • Your logs • Someone else's logs • Tracing data
- 55. Sources of data useful for debugging • Metrics • Your logs • Someone else's logs • Tracing data • Wireshark (tcpdump, libpcap)
- 58. Collecting data from RabbitMQ
- 59. Collecting data from RabbitMQ • Logs
- 60. Collecting data from RabbitMQ • Logs • rabbitmqctl status
- 61. Collecting data from RabbitMQ • Logs • rabbitmqctl status • rabbitmqctl environment
- 62. Collecting data from RabbitMQ • Logs • rabbitmqctl status • rabbitmqctl environment • rabbitmq-top (ships with RabbitMQ as of 3.6.3)
- 63. Collecting data from RabbitMQ • Logs • rabbitmqctl status • rabbitmqctl environment • rabbitmq-top (ships with RabbitMQ as of 3.6.3) • HTTP API (lots of metrics)
- 64. http://{hostname}:15672/api
- 65. curl -u guest:guest http://127.0.0.1:15672/api/overview | python -m json.tool curl -u guest:guest http://127.0.0.1:15672/api/nodes/{node} | python -m json.tool curl -u guest:guest http://127.0.0.1:15672/api/queues | python -m json.tool
- 66. Collecting data from RabbitMQ • Logs • rabbitmqctl status • rabbitmqctl environment • rabbitmq-top (ships with RabbitMQ as of 3.6.3) • HTTP API (lots of metrics) • Message tracing ("firehose")
- 67. Collecting data from RabbitMQ • HTTP API (lots of metrics) • Message tracing ("firehose") • Infrastructure metrics
- 69. Common theme?
- 70. Common theme? • Collect logs system-wide
- 71. Common theme? • Collect logs system-wide • Collect metrics system-wide
- 72. Common theme? • Collect logs system-wide • Collect metrics system-wide • Collect exceptions system-wide
- 73. Common theme? • Collect logs system-wide • Collect metrics system-wide • Collect exceptions system-wide • Trace requests (e.g. with Zipkin)
- 74. Common theme? • Collect logs system-wide • Collect metrics system-wide • Collect exceptions system-wide • Trace requests (e.g. with Zipkin) • Analyze
- 76. Common theme? • Collect logs system-wide • Collect metrics system-wide • Collect exceptions system-wide • Trace requests (e.g. with Zipkin) • Analyze • Sounds like something a structured platform can help with!
- 77. Distributed system debugging is a problem far from being solved.
- 79. Thank you
- 80. Thank you • @michaelklishin
- 81. Thank you • @michaelklishin • github.com/michaelklishin
- 82. Thank you • @michaelklishin • github.com/michaelklishin • mklishin@pivotal.io