Recommended
Recommended
More Related Content
Similar to Distributed tracing
Similar to Distributed tracing (20)
Recently uploaded
Recently uploaded (20)
Distributed tracing
- 2. 1 Story : N storytellers
- 3. What happens when you enter google.com in the browser?
- 4. Dapper Dapper Zipkin + Demo Open Tracing FAQ .. the original objective was to understand system behavior resulting from a user request to Google Whitebox Tracing - Metadata propogation
- 7. Design Goals Ubiquity Low overhead - Sampling, Adaptive Sampling, Out of band trace data collection. Application Level Transparency Scalability
- 8. Advantages Investigate overall health of system Guess which service is at fault Why which service is at fault End User - on-call engineer Bird’s eye view into overall system health Ability to drill down into a service and see why its holding up the train Long term pattern recognition
- 9. Terminologies Trace - a tree of spans - 1 for every RPC Span - has its own set of annotation including parent span Annotation - application specific data with the trace. Has a timestamp and a text value or key-value pairs
- 12. Effective access - How to effectively coalesce data in downstream systems? - Data for immediate perusal - Data for long term pattern recognition - 15 sec to reach bigtable - > 1 TB of data/day - End users want to query trace data 2 wks.
- 13. Dapper API By Trace Id Bulk Access - leverage MapReduce to provide access to billions of traces in parallel. Indexed access: composite index => lookup by service name, host name, timestamp.
- 15. Dapper Zipkin + Demo Open Tracing FAQ
- 18. Search Image Video News Translate Auth Google Image YouTube Google News Google Translate
- 19. Dapper Zipkin + Demo Open Tracing FAQ Why isn’t tracing ubiquitous? ● Tracing Instrumentation - Too hard ● Lock-in is unacceptable: instrumentation must be decoupled from vendors ● Monkey patching doesn’t scale ● Inconsistent API’ - tracing semantics must not be language dependent ● Handoff woes: tacing libs in Project X dont hand offf to tracing libs in Project Y
- 20. 1. Fractured Ecosystem 2. Tracing everything everywhere a. Interoperability is messy b. ‘Language support’ (Ruby) 3. Buy vs Build vs Adopt a. Infra Requirements & Limitations 4. Dependency Matrix of a. Tracer, Transport Layer, Collection Layer, Storage layer Challenges
- 21. OpenTracing.io OpenTracing.io Application logic µ svc frameworks Control flow packages RPC frameworks Existing instrumentation Zipkin Lightstep Jaeger Appdash Tracer X-Ray Hawkular
- 22. Buy vs Build vs Adopt
- 23. Advantages of a Service Oriented Design Collection of software services Developed by different teams Across Different platforms Different programming languages
- 24. Downsides of a Service Oriented Design Collection of software services Developed by different teams Across Different platforms Different programming languages
Editor's Notes
- A web-search example will illustrate some of the challenges such a system needs to address. A frontend service may distribute a web query to many hundreds of query servers, each searching within its own piece of the index. The query may also be sent to a number of other subsystems that may process advertisements, check spelling, or look for specialized results, including images, videos, news, and so on. Results from all of these services are selectively combined in the results page; which is called “universal search” In total, thousands of machines and many different services might be needed to process one universal search query. More- over, web-search users are sensitive to delays,