Designing Modern Streaming Data Applications

1. 1 DESIGNING MODERN STREAMING DATA APPLICATIONS ARUN KEJARIWAL, KARTHIK RAMASAMY @arun_kejariwal @karthikz

2. 2 ABOUT US

4. 4 Connected World

5. 5 Ubiquity of Real-Time Data Streams

6. 6 Data-Driven Decision Making

7. UNLOCKING INSIGHTS 7 GUIDE DECISION MAKING DETERMINING TOP TRAFFIC INTENSIVE IP ADDRESSES IDENTIFYING ARBITRAGE OPPORTUNITIES IN FINANCIAL TRAINING DETERMINING SESSIONS WHOSE DURATION >2X THAN NORMAL FORECASTING DETERIORATION IN HEALTH METRICS # UNIQUE USERS/DAY OF A MOBILE APP TARGETED ADVERTISING

8. ON-THE-FLY INSIGHTS 8 “PERISHABLE” RECENCY IS CRITICAL

9. STREAMING/FAST DATA PROCESSING 9 ✦ Events are analyzed and processed as they arrive ✦ Decisions are timely, contextual and based on fresh data ✦ Decision latency is eliminated ✦ Data in motion Ingest/ Buﬀer Analyze Act

10. MICROSERVICES MODEL INFERENCEWORKFLOWS ANALYTICS MONITORING STREAMING APPLICATION PATTERNS

11. STREAM PROCESSING PATTERN 11 ComputeMessaging Storage Data Inges6on Data Processing Results StorageData Storage Data Serving

12. LANDSCAPE PLATFORMS APPLICATION DOMAINS PRACTICAL TAKEAWAYS ANALYTICS BUSINESS IMPACT OUTLINE 12 FOCUS AREAS

13. 13 Landscape of Platforms for Streaming Data

14. PLATFORM 14 COMPONENTS PROCESSING VELOCITY STORAGE UNBOUNDED VARIETY MESSAGING VOLUME

15. 15 APACHE KAFKA RABBITMQ AKKA METAQ DATABUS ACTIVEMQ APACHE FLUME KESTREL SCRIBE APACHE PULSAR Messaging

16. NEXT GENERATION MESSAGING 16 DURABILITYCONSISTENCY EASE OF OPERATIONS ISOLATION RESILIENCY SCALABILITY KEY PROPERTIES b b b b b b INFINITE RETENTIONb

17. APACHE PULSAR 17 Flexible Messaging + Streaming System backed by a durable log storage

18. APACHE PULSAR 18 Bookie Bookie Bookie Broker Broker Broker Producer Consumer SERVING Brokers can be added independently Trafﬁc can be shifted quickly across brokers STORAGE Bookies can be added independently New bookies will ramp up trafﬁc quickly

19. APACHE PULSAR - BROKER 19 ✦ Broker is the only point of interaction for clients (producers and consumers) ✦ Brokers acquire ownership of group of topics and “serve” them ✦ Broker has no durable state ✦ Provides service discovery mechanism for client to connect to right broker

20. APACHE PULSAR - BROKER 20

21. APACHE PULSAR - CONSISTENCY 21 Bookie Bookie BookieBrokerProducer

22. APACHE PULSAR - DURABILITY 22 Bookie Bookie BookieBrokerProducer Journal Journal Journal fsync fsync fsync

23. APACHE PULSAR - ISOLATION 23

24. APACHE PULSAR - SEGMENT STORAGE 24 234…20212223…40414243…60616263… Segment 1 Segment 3 Segment 2 Segment 2 Segment 1 Segment 3 Segment 4 Segment 3 Segment 2 Segment 1 Segment 4 Segment 4

25. APACHE PULSAR - RESILIENCY 25 1234…20212223…40414243…60616263… Segment 1 Segment 3 Segment 2 Segment 2 Segment 1 Segment 3 Segment 4 Segment 3 Segment 2 Segment 1 Segment 4 Segment 4

26. APACHE PULSAR - SEAMLESS CLUSTER EXPANSION 26 1234…20212223…40414243…60616263… Segment 1 Segment 3 Segment 2 Segment 2 Segment 1 Segment 3 Segment 4 Segment 3 Segment 2 Segment 1 Segment 4 Segment 4 Segment Y Segment Z Segment X

27. APACHE PULSAR - TIERED STORAGE 27 Low Cost Storage 1234…20212223…40414243…60616263… Segment 1 Segment 3 Segment 2 Segment 2 Segment 1 Segment 3 Segment 4 Segment 3 Segment 2 Segment 1Segment 4 Segment 4

28. PARTITIONS VS SEGMENTS - WHY SHOULD YOU CARE? 28

29. PARTITIONS VS. SEGMENTS - WHY SHOULD YOU CARE? 29 ✦ In Kafka, partitions are assigned to brokers “permanently” ✦ A single partition is stored entirely in a single node ✦ Retention is limited by a single node storage capacity ✦ Failure recovery and capacity expansion require expensive “rebalancing” ✦ Rebalancing has a big impact over the system, aﬀecting regular traﬃc

30. UNIFIED MESSAGING MODEL - STREAMING 30 Pulsar topic/ partition Producer 2 Producer 1 Consumer 1 Consumer 2 Subscription A M4 M3 M2 M1 M0 M4 M3 M2 M1 M0 X Exclusive

31. UNIFIED MESSAGING MODEL - STREAMING 31 Pulsar topic/ partition Producer 2 Producer 1 Consumer 1 Consumer 2 Subscription B M4 M3 M2 M1 M0 M4 M3 M2 M1 M0 Failover In case of failure in consumer 1

32. UNIFIED MESSAGING MODEL - QUEUING 32 Pulsar topic/ partition Producer 2 Producer 1 Consumer 2 Consumer 3 Subscription C M4 M3 M2 M1 M0 Shared Traﬃc is equally distributed across consumers Consumer 1 M4M3 M2M1M0

33. DISASTER RECOVERY 33 Topic (T1) Topic (T1) Topic (T1) Subscrip6on (S1) Subscrip6on (S1) Producer (P1) Consumer (C1) Producer (P3) Producer (P2) Consumer (C2) Data Center A Data Center B Data Center C Integrated in the broker message ﬂow Simple conﬁguration to add/remove regions Asynchronous (default) and synchronous replication

34. MULTITENANCY 34 Apache Pulsar Cluster Product Safety ETL Fraud Detection Topic-1 Account History Topic-2 User Clustering Topic-1 Risk Classification MarketingCampaigns ETL Topic-1 Budgeted Spend Topic-2 Demographic Classification Topic-1 Location Resolution Data Serving Microservice Topic-1 Customer Authentication 10 TB 7 TB 5 TB 3 TB 2 TB 8 TB 4 TB 3 TB ✦ Authentication ✦ Authorization ✦ Software isolation ๏ Storage quotas, flow control, back pressure, rate limiting ✦ Hardware isolation ๏ Constrain some tenants on a subset of brokers/bookies

35. PULSAR CLIENTS 35 Apache Pulsar Cluster Java Python Go C++ C

36. PULSAR PRODUCER 36 PulsarClient client = PulsarClient.create( “http://broker.usw.example.com:8080”); Producer producer = client.createProducer( “persistent://my-property/us-west/my-namespace/my-topic”); // handles retries in case of failure producer.send("my-message".getBytes()); // Async version: producer.sendAsync("my-message".getBytes()).thenRun(() -> { // Message was persisted });

37. PULSAR CONSUMER 37 PulsarClient client = PulsarClient.create( "http://broker.usw.example.com:8080"); Consumer consumer = client.subscribe( "persistent://my-property/us-west/my-namespace/my-topic", "my-subscription-name"); while (true) { // Wait for a message Message msg = consumer.receive(); System.out.println("Received message: " + msg.getData()); // Acknowledge the message so that it can be deleted by broker consumer.acknowledge(msg); }

38. SCHEMA REGISTRY 38 ✦ Provides type safety to applications built on top of Pulsar ✦ Two approaches ๏ Client side - type safety enforcement up to the application ๏ Server side - system enforces type safety and ensures that producers and consumers remain synced ✦ Schema registry enables clients to upload data schemas on a topic basis. ✦ Schemas dictate which data types are recognized as valid for that topic

39. PULSAR SCHEMAS - HOW DO THEY WORK? 39 ✦ Enforced at the topic level ✦ Pulsar schemas consists of ๏ Name - Name refers to the topic to which the schema is applied ๏ Payload - Binary representation of the schema ๏ Schema type - JSON, Protobuf and Avro ๏ User deﬁned properties - Map of strings to strings (application speciﬁc - e.g git hash of the schema)

40. SCHEMA VERSIONING 40 PulsarClient client = PulsarClient.builder() .serviceUrl(“http://broker.usw.example.com:6650") .build() Producer<SensorReading> producer = client.newProducer(JSONSchema.of(SensorReading.class)) .topic(“sensor-data”) .sendTimeout(3, TimeUnit.SECONDS) .create() Scenario What happens No schema exists for the topic Producer is created using the given schema Schema already exists; producer connects using the same schema that’s already stored Schema is transmitted to the broker, determines that it is already stored Schema already exists; producer connects using a new schema that is compatible Schema is transmitted, compatibility determined and stored as new schema

41. HOW TO PROCESS DATA MODELED AS STREAMS 41 ✦ Consume data as it is produced (pub/sub) ✦ Light weight compute - transform and react to data as it arrives ✦ Heavy weight compute - continuous data processing ✦ Interactive query of stored streams

42. LIGHT WEIGHT COMPUTE 42 f(x) Incoming Messages Output Messages ABSTRACT VIEW OF COMPUTE REPRESENTATION

43. TRADITIONAL COMPUTE REPRESENTATION 43 DAG % % % % % Source 1 Source 2 Action Action Action Sink 1 Sink 2

44. REALIZING COMPUTATION - EXPLICIT CODE 44 public static class SplitSentence extends BaseBasicBolt { @Override public void declareOutputFields(OutputFieldsDeclarer declarer) { declarer.declare(new Fields("word")); } @Override public Map<String, Object> getComponentConfiguration() { return null; } public void execute(Tuple tuple, BasicOutputCollector basicOutputCollector) { String sentence = tuple.getStringByField("sentence"); String words[] = sentence.split(" "); for (String w : words) { basicOutputCollector.emit(new Values(w)); } } } STITCHED BY PROGRAMMERS

45. REALIZING COMPUTATION - FUNCTIONAL 45 Builder.newBuilder() .newSource(() -> StreamletUtils.randomFromList(SENTENCES)) .flatMap(sentence -> Arrays.asList(sentence.toLowerCase().split("s+"))) .reduceByKeyAndWindow(word -> word, word -> 1, WindowConfig.TumblingCountWindow(50), (x, y) -> x + y);

46. TRADITIONAL REAL TIME - SEPARATE SYSTEMS 46 Messaging Compute

47. TRADITIONAL REAL TIME SYSTEMS 47 DEVELOPER EXPERIENCE ✦ Powerful API but complicated ๏ Does everyone really need to learn functional programming? ✦ Conﬁgurable and scalable but management overhead ✦ Edge systems have resource and management constraints

48. TRADITIONAL REAL TIME SYSTEMS 48 OPERATIONAL EXPERIENCE ✦ Multiple systems to operate ๏ IoT deployments routinely have thousands of edge systems ✦ Semantic diﬀerences ๏ Mismatch and duplication between systems ๏ Creates developer and operator friction

49. LESSONS LEARNED - USE CASES 49 ✦ Data transformations ✦ Data classiﬁcation ✦ Data enrichment ✦ Data routing ✦ Data extraction and loading ✦ Real time aggregation ✦ Microservices Signiﬁcant set of processing tasks are exceedingly simple

50. EMERGENCE OF CLOUD - SERVERLESS 50 ✦ Simple function API ✦ Functions are submitted to the system ✦ Runs per events ✦ Composition APIs to do complex things ✦ Wildly popular

51. SERVERLESS VS. STREAMING 51 ✦ Both are event driven architectures ✦ Both can be used for analytics and data serving ✦ Both have composition APIs ๏ Conﬁguration based for serverless ๏ DSL based for streaming ✦ Serverless typically does not guarantee ordering ✦ Serverless is pay per action

52. STREAM NATIVE COMPUTE USING FUNCTIONS 52 ✦ Simplest possible API -function or a procedure ✦ Support for multi language ✦ Use of native API for each language ✦ Scale developers ✦ Use of message bus native concepts - input and output as topics ✦ Flexible runtime - simple standalone applications vs managed system applications APPLYING INSIGHT GAINED FROM SERVERLESS

53. PULSAR FUNCTIONS 53 SDK LESS API import java.util.function.Function; public class ExclamationFunction implements Function<String, String> { @Override public String apply(String input) { return input + "!"; } }

54. PULSAR FUNCTIONS 54 SDK API import org.apache.pulsar.functions.api.PulsarFunction; import org.apache.pulsar.functions.api.Context; public class ExclamationFunction implements PulsarFunction<String, String> { @Override public String process(String input, Context context) { return input + "!"; } }

55. PULSAR FUNCTIONS 55 ✦ Simplest possible API -function or a procedure ✦ Support for multi language ✦ Use of native API for each language ✦ Scale developers ✦ Use of message bus native concepts - input and output as topics ✦ Flexible runtime - simple standalone applications vs managed system applications

56. PULSAR FUNCTIONS 56 ✦ Function executed for every message of input topic ✦ Support for multiple topics as inputs ✦ Function output goes into output topic - can be void topic as well ✦ SerDe takes care of serialization/deserialization of messages ๏ Custom SerDe can be provided by the users ๏ Integration with schema registry

57. PROCESSING GUARANTEES 57 ✦ ATMOST_ONCE ๏ Message acked to Pulsar as soon as we receive it ✦ ATLEAST_ONCE ๏ Message acked to Pulsar after the function completes ๏ Default behavior - don’t want people to loose data ✦ EFFECTIVELY_ONCE ๏ Uses Pulsar’s inbuilt eﬀectively once semantics ✦ Controlled at runtime by user

58. DEPLOYING FUNCTIONS - BROKER 58 Broker 1 Worker Function wordcount-1 Function transform-2 Broker 1 Worker Function transform-1 Function dataroute-1 Broker 1 Worker Function wordcount-2 Function transform-3 Node 1 Node 2 Node 3

59. DEPLOYING FUNCTIONS - WORKER NODES 59 Worker Function wordcount-1 Function transform-2 Worker Function transform-1 Function dataroute-1 Worker Function wordcount-2 Function transform-3 Node 1 Node 2 Node 3 Broker 1 Broker 2 Broker 3 Node 4 Node 5 Node 6

60. DEPLOYING FUNCTIONS - KUBERNETES 60 Function wordcount-1 Function transform-1 Function transform-3 Pod 1 Pod 2 Pod 3 Broker 1 Broker 2 Broker 3 Pod 7 Pod 8 Pod 9 Function dataroute-1 Function wordcount-2 Function transform-2 Pod 4 Pod 5 Pod 6

61. BUILT-IN STATE MANAGEMENT IN FUNCTIONS 61 ✦ Functions can store state in inbuilt storage ๏ Framework provides a simple library to store and retrieve state ✦ Support server side operations like counters ✦ Simpliﬁed application development ๏ No need to standup an extra system

62. DISTRIBUTED STATE IN FUNCTIONS 62 import org.apache.pulsar.functions.api.Context; import org.apache.pulsar.functions.api.PulsarFunction; public class CounterFunction implements PulsarFunction<String, Void> { @Override public Void process(String input, Context context) throws Exception { for (String word : input.split(".")) { context.incrCounter(word, 1); } return null; } }

63. PULSAR - DATA IN AND OUT 63 ✦ Users can write custom code using Pulsar producer and consumer API ✦ Challenges ๏ Where should the application to publish data or consume data from Pulsar? ๏ How should the application to publish data or consume data from Pulsar? ✦ Current systems have no organized and fault tolerant way to run applications that ingress and egress data from and to external systems

64. PULSAR IO TO THE RESCUE 64 Apache Pulsar ClusterSource Sink

65. INTERACTIVE QUERYING OF STREAMS - PULSAR SQL 65 1234…20212223…40414243…60616263… Segment 1 Segment 3 Segment 2 Segment 2 Segment 1 Segment 3 Segment 4 Segment 3 Segment 2 Segment 1 Segment 4 Segment 4 Segment Reader Segment Reader Segment Reader Segment Reader Coordinator

66. PULSAR IO - EXECUTION 66 Broker 1 Worker Sink Cassandra-1 Source Kinesis-2 Broker 2 Worker Source Kinesis-1 Source Twitter-1 Broker 3 Worker Sink Cassandra-2 Source Kinesis-3 Node 1 Node 2 Node 3 Fault tolerance Parallelism Elasticity Load Balancing On-demand updates

67. EASE OF OPERATIONS 67 ✦ Brokers don’t have durable state ๏ Easily replaceable ๏ Topics are immediately reassigned to healthy brokers ✦ Expanding capacity ๏ Simply add new broker node ๏ If other brokers are overloaded, traﬃc will be automatically assigned ✦ Load manager ๏ Monitor traﬃc load on all brokers (CPU, memory, network, topics) ๏ Initially place topics to least loaded brokers ๏ Reassign topics when a broker is overloaded

68. CONSUMER BACKLOG 68 ✦ Metrics are available to make assessments ๏ When a problem started ๏ How big is backlog? messages? disk space? ๏ How fast is draining? ๏ What’s the ETA to catch up with publishers? ✦ Establish where is the bottleneck ๏ Application is not fast enough ๏ Disk read I/O

69. ENFORCING MULTITENANCY 69 ✦ Ensure tenants don’t cause performance issues on other tenants ๏ Backlog quotas ๏ Soft isolation - ﬂow control and throttling ✦ In cases when user behavior is triggering performance degradation ๏ Hard isolation as a last resource for quick reaction while proper ﬁx is deployed ๏ Isolate tenant on a subset of brokers ๏ Can be also applied at the storage node level

70. PULSAR PERFORMANCE 70

71. PULSAR PERFORMANCE - LATENCY 71

72. APACHE PULSAR VS. APACHE KAFKA 72 Mul$-tenancy A single cluster can support many tenants and use cases Seamless Cluster Expansion Expand the cluster without any down $me High throughput & Low Latency Can reach 1.8 M messages/s in a single par$$on and publish latency of 5ms at 99pct Durability Data replicated and synced to disk Geo-replica$on Out of box support for geographically distributed applica$ons Uniﬁed messaging model Support both Topic & Queue seman$c in a single model Tiered Storage Hot/warm data for real $me access and cold event data in cheaper storage Pulsar Func$ons Flexible light weight compute Highly scalable Can support millio

Designing Modern Streaming Data Applications

Designing Modern Streaming Data Applications

Recommended

More Related Content

What's hot

What's hot(20)

Similar to Designing Modern Streaming Data Applications

Similar to Designing Modern Streaming Data Applications(20)

More from Arun Kejariwal

More from Arun Kejariwal(20)

Recently uploaded

Recently uploaded(20)

Designing Modern Streaming Data Applications