Making Sense of Your Event-Driven Dataflows (Jorge Esteban Quilcate Otoya, SYSCO AS) Kafka Summit NYC 2019
•
2 likes•1,008 views
Contrary to RPC-like applications, where communication and dependencies are explicitly defined between Services; data flowing between Event-Driven Applications is defined by how do they react to and emit events. A trade-off between Data-flow explicitness and Service autonomy becomes apparent between this two architectural-styles. The goal in this presentation is to demonstrate how Distributed-Tracing can help to cope with this trade-off, turning messaging exchange between decoupled, autonomous, Event-Driven Services, into explicit Data-flows. Zipkin project brings a Distributed-Tracing infrastructure that enables the collection, processing, and visualization of traces produced by RPC-based, as well as messaging-based applications. This presentation includes demonstrations on how to enable Tracing for Kafka Streams applications, Kafka Connectors, and KSQL; evidencing how implicit Services behavior and communication through the event-log become can become explicit via Distributed-Tracing. But collecting and visualizing traces is just the first step. In order to create insights from tracing-data, models has to be built to enable an better understanding from the system, and improve our operational capabilities. Including research-based experiences from Netflix[1] and Facebook[2] on how tracing-data has been processed and polished with multiple purposes, this presentation will cover how service-dependency analysis and anomaly-detection models can be built on top of it.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Making Sense of Your Event-Driven Dataflows (Jorge Esteban Quilcate Otoya, SYSCO AS) Kafka Summit NYC 2019
Similar to Making Sense of Your Event-Driven Dataflows (Jorge Esteban Quilcate Otoya, SYSCO AS) Kafka Summit NYC 2019 (20)
More from confluent
More from confluent (20)
Recently uploaded
Recently uploaded (20)
Making Sense of Your Event-Driven Dataflows (Jorge Esteban Quilcate Otoya, SYSCO AS) Kafka Summit NYC 2019
- 1. @jeqo89 at #kafkasummit Making sense of event-driven dataflows Distributed tracing for Apache Ka a®-based applications with Zipkin
- 11. @jeqo89 at #kafkasummit Complexity happens…
- 12. @jeqo89 at #kafkasummit “Complexity is anything [...] that makes a system hard to understand and modify” John Ousterhout, “A Philosophy of Software Design”
- 13. @jeqo89 at #kafkasummit “Complexity is caused by two things: dependencies and obscurity” John Ousterhout, “A Philosophy of Software Design”
- 15. @jeqo89 at #kafkasummit Jorge Esteban Quilcate Otoya twitter: @jeqo89 | github: jeqo Peruvian in Oslo, Norway Integration team at SYSCO AS Part of Apache Kafka and Zipkin communities
- 16. @jeqo89 at #kafkasummit Talk: “Making sense of your event-driven dataflows” 40 min Q&AWhy? What distributed tracing? How to instrument Kafka apps? Demo What’s next? Demo time
- 18. @jeqo89 at #kafkasummit “Demystifying” Kafka client configurations Kafka producers Kafka Streams Kafka Consumer github.com/jeqo/tracing-kafka-apps
- 19. @jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps Trace ID Trace metrics Trace timeline Spans
- 22. @jeqo89 at #kafkasummit Is Kafka producer `send` sync or async? github.com/jeqo/tracing-kafka-apps Blocking call
- 23. @jeqo89 at #kafkasummit github.com/jeqo/tracing-kafka-apps Non-blocking call Is Kafka producer `send` sync or async?
- 28. @jeqo89 at #kafkasummit “The more accurately you try to measure the position of a particle, the less accurately you can measure its speed” Heisenberg's uncertainty principle
- 29. @jeqo89 at #kafkasummit CLIENT SERVER TraceContext=abc tracer tracer Traces reporting: Annotation-based approach TRACES
- 30. @jeqo89 at #kafkasummit PRODUCER CONSUMER tracer tracerTraceContext=abc BROKER TraceContext=abc TRACES
- 31. @jeqo89 at #kafkasummit /** Annotation-based approach **/ ScopedSpan span = tracer.startScopedSpan("process"); try { // The span is in "scope" doProcess(); } catch (RuntimeException | Error e) { span.error(e); // mark as error throw e; } finally { span.finish(); // always finish }
- 32. @jeqo89 at #kafkasummit /** Annotation-based approach **/ ScopedSpan span = tracer.startScopedSpan("process"); try { // The span is in "scope" doProcess(); } catch (RuntimeException | Error e) { span.error(e); // mark as error throw e; } finally { span.finish(); // always finish }
- 33. @jeqo89 at #kafkasummit /** Annotation-based approach **/ ScopedSpan span = tracer.startScopedSpan("process"); try { // The span is in "scope" doProcess(); } catch (RuntimeException | Error e) { span.error(e); // mark as error throw e; } finally { span.finish(); // always finish }
- 34. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ Producer<K, V> producer = new KafkaProducer<>(settings); Producer<K, V> tracedProducer = kafkaTracing.producer(producer); producer.send( new ProducerRecord<>( "my-topic", key, value ));
- 35. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ Producer<K, V> producer = new KafkaProducer<>(settings); Producer<K, V> tracedProducer = kafkaTracing.producer(producer); // wrap tracedProducer.send( new ProducerRecord<>( "my-topic", key, value ));
- 36. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ Consumer<K, V> consumer = new KafkaConsumer<>(settings); Consumer<K, V> tracedConsumer = kafkaTracing.consumer(consumer); while (running) { var records = consumer.poll(1000); records.forEach(this::process); }
- 37. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ Consumer<K, V> consumer = new KafkaConsumer<>(settings); Consumer<K, V> tracedConsumer = kafkaTracing.consumer(consumer); // wrap while (running) { var records = tracedConsumer.poll(1000); records.forEach(this::process); }
- 38. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Clients **/ void process(ConsumerRecord<K, V> record){ // extract span from record headers Span span = kafkaTracing.nextSpan(record) .name("process") .start(); try (var ws = tracer.withSpanInScope(span)) { doProcess(record); } catch (RuntimeException | Error e) { span.error(e); throw e; } finally { span.finish(); } }
- 39. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Streams **/ var b = new StreamsBuilder(); b.stream("input-topic") .map(this::parseRecord)) .join(table, this::tableJoiner) .transformValues(this::transform)) .to("output-topic"); KafkaStreams kafkaStreams = new KafkaStreams(b.build(), config); kafkaStreams.start();
- 40. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Streams **/ var b = new StreamsBuilder(); b.stream("input-topic") .map(this::parseRecord)) .join(table, this::tableJoiner) .transformValues(this::transform)) .to("output-topic"); KafkaStreams kafkaStreams = // wrap ksTracing.kafkaStreams(b.build(), config); kafkaStreams.start();
- 41. @jeqo89 at #kafkasummit /** Instrumentation for Kafka Streams **/ var b = new StreamsBuilder(); b.stream("input-topic") .transform(ksTracing.map(“parse”, this::parseRecord)) .join(table, this::tableJoiner) .transformValues(ksTracing.transformValues( “transform”, this::transform))) .to("output-topic");
- 42. @jeqo89 at #kafkasummit CLIENT SERVER TraceContext=abc Traces reporting: Black-box approach agent agent TRACES
- 43. @jeqo89 at #kafkasummit CLIENT SERVER TraceContext=abc Traces reporting: mixed approach agent agent TRACES tracer tracer
- 44. @jeqo89 at #kafkasummit Report Transport services
- 45. @jeqo89 at #kafkasummit /** Transports for Zipkin **/ var sender = URLConnectionSender.create( "http://localhost:9411/api/v2/spans" ); var reporter = AsyncReporter.create(sender);
- 46. @jeqo89 at #kafkasummit /** Transports for Zipkin **/ var sender = KafkaSender.newBuilder() .bootstrapServers( "localhost:9092") .build(); var reporter = AsyncReporter.create(sender);
- 47. @jeqo89 at #kafkasummit /** Transports for Zipkin **/ var sender = BringYourOwnSender.newBuilder() .build(); var reporter = AsyncReporter.create(sender);
- 48. @jeqo89 at #kafkasummit Report Transport Storage BringYourOwnDB services
- 49. @jeqo89 at #kafkasummit Report Transport Storage Dependencies (batch) services Data-at-rest
- 50. @jeqo89 at #kafkasummit Streaming Messaging Kafka Clients REST Proxy KSQL Kafka Source Connector Kafka Streams Kafka Sink Connector
- 51. @jeqo89 at #kafkasummit REST Proxy KSQL Kafka Source Connector Kafka Sink Connector Kafka Interceptors
- 52. @jeqo89 at #kafkasummitProducer Interceptor API
- 53. @jeqo89 at #kafkasummitConsumer Interceptor API
- 54. @jeqo89 at #kafkasummitDemo: Tracing Kafka-based applications github.com/jeqo/talk-kafka-zipkin
- 55. @jeqo89 at #kafkasummit What’s next?
- 56. @jeqo89 at #kafkasummit Report Transport Storage Dependencies (batch) Distributed Tracing IS A Stream Processing Problem Data-at-rest services
- 57. @jeqo89 at #kafkasummit Distributed Tracing IS A Stream Processing Problem Span Consumer Trace Aggregation Span Store spans-collected traces-completed Dependencies Store github.com/jeqo/zipkin-storage-kafka Custom processors
- 60. @jeqo89 at #kafkasummitCanopy: An End-to-End Performance Tracing And Analysis System”
- 65. @jeqo89 at #kafkasummit Peter Alvaro et al., “Automating Failure Testing Research at Internet Scale”
- 66. @jeqo89 at #kafkasummit is there anyone applying this?
- 67. @jeqo89 at #kafkasummit Haystack: tracing and analysis platform Tracing Trends and Metrics Anomaly Detection Remediation Alerting services
- 68. @jeqo89 at #kafkasummit Demo: Extending Zipkin with Kafka and Haystack github.com/jeqo/talk-kafka-zipkin
- 69. @jeqo89 at #kafkasummit * Demo 1, source code: github.com/jeqo/tracing-kafka-apps * Demo 2, source code: github.com/jeqo/talk-kafka-zipkin * Blog post: confluent.io/blog/importance-of-distributed-tracing-for-apache-kafka-based-applications * Zipkin: github.com/openzipkin (moving to Apache foundation), gitter.im/openzipkin/zipkin * Sites using Zipkin: cwiki.apache.org/confluence/display/ZIPKIN/Sites * Haystack: github.com/ExpediaDotCom/haystack, gitter.im/expedia-haystack * Zipkin Kafka Backend: github.com/jeqo/zipkin-storage-kafka * Kafka Interceptor for Zipkin: github.com/sysco-middleware/kafka-interceptor-zipkin * Martin Kleppmann et al. 2019. Online Event Processing. https://dl.acm.org/citation.cfm?id=3321612 * John Ousterhout. A Philosophy of Software Design. www.amazon.com/Philosophy-Software-Design-John-Ousterhout/dp/1732102201 * Jonathan Kaldor et al.2017. Canopy: An End-to-End Performance Tracing And Analysis System.SOSP’17(2017). doi.org/10.1145/3132747.3132749 * Peter Alvaro et al.2016. Automating Failure Testing Research at Internet Scale.SoCC ’16. dx.doi.org/10.1145/2987550.2987555 Resources