Download as PDF, PPTX
Uploaded byFlink Forward
Flink Forward Berlin 2018: Brian Wolfe - "Upshot: distributed tracing using Flink"
The document discusses the implementation of distributed tracing at Airbnb using Flink to enhance observability for software engineers. It highlights successes, such as processing 3 million events per second and achieving better performance, alongside challenges related to data coupling and serialization. Future steps include optimizing performance impact, managing load, and enhancing connectivity between observability tools.
More Related Content
Similar to Flink Forward Berlin 2018: Brian Wolfe - "Upshot: distributed tracing using Flink"
Flink Forward Berlin 2018: Brian Wolfe - "Upshot: distributed tracing using Flink"
- 1. Upshot: Distributed Tracing withFlink Brian Wolfe / 2018-09-05 / Flink Forward 2018
- 2. Tracing with Flink ●Observability at Airbnb ● Trace reconstruction job ● Successes (and failures)
- 3. Tracing with Flink ●Observability at Airbnb ● Trace reconstruction job ● Successes (and failures)
- 5. Airbnb 5 M + Listingsworldwide 81,000 Cities with listings 191+ Countries with listings 300 million+ Total guest arrivals
- 6.
- 7. Observability Ensure that softwareengineers at Airbnb have the monitoring and introspection tools to successfully operate and develop their services.
- 8.
- 9. Reality 1: manytools
- 10. Reality 1: manytools ?
- 11. Reality 1: manytools
- 12. Reality 2: datasilos
- 13. Reality 3: choosingthe right data is hard Present Enrich DS Validate
- 14. Reality 3: choosingthe right data is hard Present Enrich DS Validate Late error in request
- 15. Reality 3: choosingthe right data is hard Present Enrich DS Validate Late error in request Give me ALL the data!!
- 16. Tracing with Flink ●Observability at Airbnb ● Trace reconstruction job ● Successes (and failures)
- 17.
- 18.
- 19. Flink Aggregation :CollectTrace After one event
- 20. Flink Aggregation :CollectTrace After more events
- 21. Flink Aggregation :CollectTrace After all events
- 22. Flink Aggregation :CollectTrace After all events
- 23. Tracing with Flink ●Observability at Airbnb ● Trace reconstruction job ● Successes (and failures)
- 24. Success: it works 3Million Events / s
- 25. Success: One Instrumentation,Several Uses High dynamic range histograms in DruidIndividual request visualizations ResponseTime(ms) Quantilems Servicecall
- 27. Success : Post-Samplingof Events
- 28. Success: testing bydual-reading
- 29.
- 30. Problem: serialization performance 3xperformance improvement with a custom serializer
- 31. Problem: serialization performance 3xperformance improvement with a custom serializer String deserialization, alloc, GC is still > 25% of CPU cost.
- 32. Problem: coupling betweenall operators
- 33. Problem: coupling betweenall operators
- 34. Problem: coupling betweenall operators
- 35. Problem: coupling betweenall operators
- 36. Problem: coupling betweenall operators
- 37. Problem: coupling betweenall operators
- 38. Problem: coupling betweenall operators
- 39. Problem: coupling betweenall operators
- 40. Problem: coupling betweenall operators
- 41. Problem: coupling betweenall operators Better after Flink 1.5, but still a problem
- 42.
- 43. Problem: Coupling BetweenAll OperatorsUseFlinkpartitionerto writetoKafka
- 44. Problem: Coupling BetweenAll OperatorsUseFlinkpartitionerto writetoKafka Limit fan-out
- 45.
- 46. Next Steps ● Localizeimpact of degraded performance ● Load shed when necessary ● Add more links between observability tools
- 47. Observability Team Willie Yao NelsonGauthier Jerry Chung Rong Hu Chen Luo Sarah Wada Harry Shoff Nathan Baxter Joseph Sofaer