Download as PDF, PPTX
Uploaded byTyler Treat
Cloud-Native Observability
The document discusses cloud-native observability, emphasizing the importance of structured logging and data pipelines for effective monitoring and debugging in complex systems. It outlines challenges such as accessing observability data across different platforms, the necessity of standard specifications for data collection, and the implementation of an observability pipeline to handle data efficiently. Key components include structured logging, data collectors, and routers, which facilitate the seamless transfer and analysis of system behavior to enhance operational reliability.
More Related Content
Similar to Cloud-Native Observability
Cloud-Native Observability
- 1. @tyler_treat Cloud-Native Observability Tyler Treat/ Cloud Native - Madison / June 6, 2019
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10. @tyler_treat APM Debugger Profiler SSH Testing in Productionat Scale, Amit Gud grep
- 11.
- 12.
- 13.
- 14.
- 15. @tyler_treat Data Available Understanding Known Knowns •Things we are aware of and understand • “The system has a 1GB memory limit”
- 16. @tyler_treat Data Available Understanding Known Knowns •Things we are aware of and understand • “The system has a 1GB memory limit” Known Unknowns • Things we are aware of but don’t understand • “The system exceeded its memory limit and crashed, causing an outage”
- 17. @tyler_treat Data Available Understanding Unknown Knowns •Things we understand but are not aware of • “We implemented an orchestrator to ensure the system is always running” Known Knowns • Things we are aware of and understand • “The system has a 1GB memory limit” Known Unknowns • Things we are aware of but don’t understand • “The system exceeded its memory limit and crashed, causing an outage”
- 18. @tyler_treat Data Available Understanding Unknown Knowns •Things we understand but are not aware of • “We implemented an orchestrator to ensure the system is always running” Known Knowns • Things we are aware of and understand • “The system has a 1GB memory limit” Unknown Unknowns • Things we are neither aware of nor understand • “Instances churn because the orchestrator restarts the process when it approaches its memory limit, causing sporadic failures and slowdowns” Known Unknowns • Things we are aware of but don’t understand • “The system exceeded its memory limit and crashed, causing an outage”
- 19. @tyler_treat Data Available Understanding Unknown Knowns •Things we understand but are not aware of • “We implemented an orchestrator to ensure the system is always running” Known Knowns • Things we are aware of and understand • “The system has a 1GB memory limit” Unknown Unknowns • Things we are neither aware of nor understand • “Instances churn because the orchestrator restarts the process when it approaches its memory limit, causing sporadic failures and slowdowns” Known Unknowns • Things we are aware of but don’t understand • “The system exceeded its memory limit and crashed, causing an outage” FACTS
- 20. @tyler_treat Data Available Understanding Unknown Knowns •Things we understand but are not aware of • “We implemented an orchestrator to ensure the system is always running” Known Knowns • Things we are aware of and understand • “The system has a 1GB memory limit” Unknown Unknowns • Things we are neither aware of nor understand • “Instances churn because the orchestrator restarts the process when it approaches its memory limit, causing sporadic failures and slowdowns” Known Unknowns • Things we are aware of but don’t understand • “The system exceeded its memory limit and crashed, causing an outage” FACTS HYPOTHESES
- 21. @tyler_treat Data Available Understanding Unknown Knowns •Things we understand but are not aware of • “We implemented an orchestrator to ensure the system is always running” Known Knowns • Things we are aware of and understand • “The system has a 1GB memory limit” Unknown Unknowns • Things we are neither aware of nor understand • “Instances churn because the orchestrator restarts the process when it approaches its memory limit, causing sporadic failures and slowdowns” Known Unknowns • Things we are aware of but don’t understand • “The system exceeded its memory limit and crashed, causing an outage” ASSUMPTIONS FACTS HYPOTHESES
- 22. @tyler_treat Unknown Unknowns • Thingswe are neither aware of nor understand • “Instances churn because the orchestrator restarts the process when it approaches its memory limit, causing sporadic failures and slowdowns” DISCOVERIES Data Available Understanding Unknown Knowns • Things we understand but are not aware of • “We implemented an orchestrator to ensure the system is always running” Known Knowns • Things we are aware of and understand • “The system has a 1GB memory limit” Known Unknowns • Things we are aware of but don’t understand • “The system exceeded its memory limit and crashed, causing an outage” ASSUMPTIONS FACTS HYPOTHESES
- 23. @tyler_treat Unknown Unknowns • Thingswe are neither aware of nor understand • “Instances churn because the orchestrator restarts the process when it approaches its memory limit, causing sporadic failures and slowdowns” DISCOVERIES Data Available Understanding Known Unknowns • Things we are aware of but don’t understand • “The system exceeded its memory limit and crashed, causing an outage” HYPOTHESES MonitoringObservability
- 24. @tyler_treat Unknown Unknowns • Thingswe are neither aware of nor understand • “Instances churn because the orchestrator restarts the process when it approaches its memory limit, causing sporadic failures and slowdowns” DISCOVERIES Data Available Understanding Known Unknowns • Things we are aware of but don’t understand • “The system exceeded its memory limit and crashed, causing an outage” HYPOTHESES TestingExploring
- 25. @tyler_treat Observability Data application logs systemlogs audit logs application metrics distributed traces events
- 26. @tyler_treat Some challenges… Observability Data application logs systemlogs audit logs application metrics distributed traces events - Locked up inside a single vendor’s solution - Not readily available across the enterprise (or in some cases, too readily available) - Many tools and products needed for different data and use cases - Tool and data needs vary from team to team - Ever-changing landscape of tools, products, and services - Sheer volume of data can be overwhelming
- 27. @tyler_treat System Splunk Universal Forwarder Datadog APM Agent Universal Analytics Client AmazonGlacier S3 Client … Datadog Metrics Agent
- 28. System Splunk Universal Forwarder Datadog APM Agent Universal Analytics Client S3Client … Datadog Metrics Agent System Splunk Universal Forwarder Datadog APM Agent Universal Analytics Client S3 Client … Datadog Metrics Agent System Splunk Universal Forwarder Datadog APM Agent Universal Analytics Client S3 Client … Datadog Metrics Agent System Sp Un For Datad A Universal Analytics Client S3 Client … Datado A System Splunk Universal Forwarder Datadog APM Agent Universal Analytics Client S3 Client … Datadog Metrics Agent System Splunk Universal Forwarder Datadog APM Agent Universal Analytics Client S3 Client … Datadog Metrics Agent System Splunk Universal Forwarder Datadog APM Agent Universal Analytics Client S3 Client … Datadog Metrics Agent System Sp Un For Datad A Universal Analytics Client S3 Client … Datado A Splunk Universal Forwarder Universal Analytics Client Splunk Universal Forwarder Universal Analytics Client Splunk Universal Forwarder Universal Analytics Client Sp Un For Universal Analytics Client System System System System
- 29. @tyler_treat How big ofa lift is it for your organization to change tools?
- 30. @tyler_treat How easy isit to experiment with new ones?
- 31. @tyler_treat Data Sources • VMs •Containers • Load balancers • Service meshes • Audit logs • VPC flow logs • Firewall logs • … Data Sinks • Centralized logging • SIEM • Monitoring • APM • Alerting • Cold storage • BI • … What data to send? Where to send it? How to send it?
- 32.
- 33. @tyler_treat What data tosend? Where to send it? How to send it? Data Sources • VMs • Containers • Load balancers • Service meshes • Audit logs • VPC flow logs • Firewall logs • … Data Sinks • Centralized logging • SIEM • Monitoring • APM • Alerting • Cold storage • BI • … Observability Pipeline
- 34.
- 35. @tyler_treat Structure your damndata. 1. Data Specifications
- 36. @tyler_treat log.error(“User '{}' loginfailed”.format(user))
- 37. @tyler_treat ERROR 2019-04-05 13:26.42User ‘tylertreat' login failed
- 38.
- 39. @tyler_treat { “timestamp”: “2019-04-05 13:26.42”, “level”:“ERROR”, “event”: “user_login_error”, “user”: “tylertreat”, “email”: “tyler.treat@realkinetic.com”, “error”: “Invalid username or password”, “message”: “User login failed” }
- 40.
- 41. @tyler_treat Pass a contextobject to everything.
- 42. @tyler_treat def login(ctx, username,email, password): ctx.set(user=username, email=email) ... log.error(“User login failed”, event=LOGIN_ERROR, context=ctx, error=error) ...
- 43. @tyler_treat { “timestamp”: “2019-04-05 13:26.42”, “level”:“ERROR”, “event”: “user_login_error”, “context”: { “id”: “accfbb8315c44a52ad893ca6772e1caf”, “http_method”: “POST”, “http_path”: “/login”, “user”: “tylertreat”, “email”: “tyler.treat@realkinetic.com”, }, “error”: “Invalid username or password”, “message”: “User login failed” }
- 44. @tyler_treat { “timestamp”: “2019-04-05 13:26.42”, “level”:“ERROR”, “event”: “user_login_error”, “context”: { “id”: “accfbb8315c44a52ad893ca6772e1caf”, “http_method”: “POST”, “http_path”: “/login”, “user”: “tylertreat”, “email”: “tyler.treat@realkinetic.com”, }, “error”: “Invalid username or password”, “message”: “User login failed” }
- 45. @tyler_treat Create standard specsfor each data type collected (logs, metrics, traces).
- 46. @tyler_treat Specs can enforcerequired fields (e.g. user id, license, trace id) and data types.
- 47. @tyler_treat { “timestamp”: “2019-04-05 13:26.42”, “level”:“INFO”, “event”: “user_login”, “context”: { “id”: “accfbb8315c44a52ad893ca6772e1caf”, “http_method”: “POST”, “http_path”: “/login”, “user”: “tylertreat”, “user_id”: “3bb12f6c63274abe87fd1ee4ee37f3d2”, “license”: “942e6543f0844be680e72003d5e060fd”, “email”: “tyler.treat@realkinetic.com”, } }
- 48. @tyler_treat Specs alone aren’tenough! 2. Specification Libraries
- 49.
- 50. @tyler_treat • Java: log4j •Go: logrus • Python: structlog • Ruby: ruby-cabin • .NET: serilog • JS: structured-log • etc. There are many existing libraries for structured logging.
- 51. @tyler_treat For tracing and metrics,there are vendor-neutral APIs like OpenTracing and OpenCensus.
- 52. @tyler_treat We need alightweight agent that can collect data from hosts/containers. 3. Data Collector
- 53. @tyler_treat Collect data, performtransformations/ filters, and write it to the data pipeline.
- 54. @tyler_treat Typically runs asan agent on the host (DaemonSet in Kubernetes).
- 55. @tyler_treat Data is writtento stdout/stderr or a Unix domain socket.
- 56.
- 57. @tyler_treat We need ascalable, fault-tolerant data stream to handle the firehose of observability data generated. 4. Data Pipeline
- 58. @tyler_treat This also providesa buffer that decouples producers from consumers.
- 59.
- 60.
- 61. @tyler_treat We need acomponent to consume data from the pipeline, perform filtering, and write it to the appropriate backends. 5. Data Router
- 62. @tyler_treat This is wherethe data spec comes into play.
- 63. @tyler_treat The data shapedetermines how incoming data is handled.
- 64. @tyler_treat Data Pipeline Amazon Glacier DataRouter logs traces metrics
- 65. @tyler_treat Data Pipeline Amazon Glacier DataRouter logs traces metrics
- 66. @tyler_treat Data Pipeline Amazon Glacier DataRouter logs traces metrics
- 67. @tyler_treat This is primarilya stateless component writing to APIs.
- 68.
- 69. @tyler_treat Piecing It AllTogether
- 70.
- 71. @tyler_treat You don’t needto build it out all in one go.
- 72. @tyler_treat There are quickwins along the way!
- 73. @tyler_treat Evolving to anObservability Pipeline • Adopt structured logging • Move log/data collection out of process • Use a centralized logging system • Introduce a streaming data solution • Start adding data consumers
- 74.
- 75.
- 76.
- 77.
- 78.
- 79.
- 80. @tyler_treat CI/CD Pre- Production (theorizing about known unknowns) Post- Production (learningfrom unknown unknowns) Observability
- 81.
- 82. @tyler_treat Trip Service Flight Service HotelService Car Rental ServiceDynamoDB DynamoDB DynamoDB DynamoDB Book Trip
- 83.
- 84.
- 85. @tyler_treat And now here’ssome YAML…
- 86.
- 87.
- 88.
- 89.
- 90.
- 91.
- 92.
- 93.
- 94.
- 95.