Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Data	Guarantees	and	Fault	Tolerance	in	
Streaming	Systems
Roshan	Nai...
2 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Types	of	Realtime	Streaming	systems
à Data	Movement/Ingestion
– Coll...
3 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Data	Handling	Guarantees
à At	Most	Once
– Message	loss	is	possible.	...
4 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Common	Problems	in	Streaming
à Source/Destination	Service	not	availa...
5 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Streaming	Pipelines
6 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Streaming	Pipeline
webservers
Flume/Nifi Kafka
Storm
Hive
7 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Simple	Streaming	to	HDFS
8 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Problems	of	Streaming	directly	to	HDFS
à Partial	writes	(due	to	conn...
9 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Flume
10 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Flume	Components
Agent: Flume	Process.	Handles	data	movement.
Sourc...
11 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Simple	Flume	data	flow
12 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Topologies	- Aggregation
13 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Flume	Transactions
txnA txnB
txnA
txnB
txnC
txnD
txnE
txnF
14 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Flume	Fault	tolerance	and	Guarantees
à Guarantees:
– Depends	on	the...
15 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Hive	Streaming	APIs
https://cwiki.apache.org/confluence/display/Hiv...
16 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Problems	of	streaming	directly	to	HDFS
à File	corruption	due	to	par...
17 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Hive	Streaming	APIs
à First	class	support	for	ingesting	data	into	H...
18 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Hive	Streaming	APIs	- Pseudo	Code
hiveEP = new HiveEndPoint(metatst...
19 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Hive	Streaming	APIs	– Guarantees
à Guarantees:
– At	least	once
20 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Kafka
21 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Kafka	– Fault	tolerance
à Brokers :	host	messages	on	local	disk	for...
22 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Kafka	– Data	Guarantees
à Producers	API	:	Used	by	client	processes	...
23 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Kafka	– Fault	tolerance	&	Guarantees
à Consumer	API	:	Used	by	clien...
24 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Kafka	– Fault	tolerance	&	Guarantees
à Read	from	Kafka		-->		Write	...
25 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Storm
26 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Storm	– Basics
à Distributed	Realtime	computing	engine.	
à Allows	c...
27 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Storm	– Data	Guarantees
à Three	modes	of	usage
– Without	ACK-ing: A...
28 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Flink
29 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Flink – Basics
à Distributed	Realtime	&	Batch	computing	engine.
à C...
30 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Flink – Checkpointing
à Periodically	takes	a	distributed	snapshot	o...
31 ©	Hortonworks	Inc.	2011	– 2016.	All	Rights	Reserved
Thank	You	!
Questions	?
Upcoming SlideShare
Loading in …5
×

Data Guarantees and Fault Tolerance in Streaming Systems

711 views

Published on

Does your big data streaming pipeline have a hole in its pocket ? Streaming involves gathering data, processing it and delivering the results to the intended destinations in real time. Glitches at any stage can cause data loss unless the products employed in the pipeline provide the necessary guarantees and are configured properly to deliver on those guarantees.

Realtime stream processing brings unique challenges with respect to data handling guarantees and fault tolerance. Each streaming product comes with a unique approach to tackle these problems. When assembling a streaming pipeline, it is important to understand this critical topic for proper selection and configuration of the individual components of the pipeline. The exercise to determine if you are missing some records in your data lake can be expensive, but it can be extremely difficult to track down the cause to prevent it from recurring.

To help you build reliable streaming pipelines, this talk will give you a better understanding of the problems involved in realtime streaming, the kinds of guarantees involved and how they are handled in popular open source products such as Storm, Flink, Kafka, Hive Streaming APIs and Flume.

Speaker
Roshan Naik, Senior MTS, Hortonworks

Published in: Technology
  • Be the first to comment

Data Guarantees and Fault Tolerance in Streaming Systems

  1. 1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Data Guarantees and Fault Tolerance in Streaming Systems Roshan Naik, Hortonworks Dataworks Summit Sept 21st 2017, San Jose
  2. 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Types of Realtime Streaming systems à Data Movement/Ingestion – Collect data (messages/files) from multiple sources (webservers, sensors, etc.) and aggregate the data into locations like Kafka or HDFS – May support simpler kinds of data processing (transforms, filtering) – E.g: Flume, Nifi à Stream Computing Engines – Usually fed data from aggregation points (like Kafka and HDFS) – Support complex data processing (aggregations, joins, windowing, etc.) – E.g: Storm, Flink à Aggregation Points / Stores – Intermediate or final destination for streaming data. – E.g: Kafka, Hive, Hdfs
  3. 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Data Handling Guarantees à At Most Once – Message loss is possible. No duplication of messages. – In failure scenarios, events will be dropped and re-delivery attempts are not made. à At Least Once – No message loss. But duplication is possible. – In event of failure scenarios, re-delivery attempts are made. à Exactly Once – No message loss. No duplication of messages. – Typically requires the use of deduplication or idempotent updates.
  4. 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Common Problems in Streaming à Source/Destination Service not available – Intermittent or Extended Outages – Solution: Periodically retry to see if situation has changed à Unable to keep up with increase in traffic – Destination may not able to handle higher traffic – Streaming system does not have enough capacity to handle higher throughput – Solution: Add more machines to handle the capacity à Processes crashing – Streaming system’s processes crashing means loss of un-persisted data or state – Solution: Restart processes, persistent state and ability to rewind and replay à Hardware outages – Loss of data persisted to local disk as well as anything in-memory. – Solution: Replicate data, prefer remote location over local disk.
  5. 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Streaming Pipelines
  6. 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Streaming Pipeline webservers Flume/Nifi Kafka Storm Hive
  7. 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Simple Streaming to HDFS
  8. 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Problems of Streaming directly to HDFS à Partial writes (due to connection failures) – Depending on file format, data in it may not be recoverable à Need to ensure concurrent writers don t write to same file à Periodic File rotation à Compacting small files into larger – Requires a separate background process that runs perdiodically. à Marking files as ‘done’ so that it can be queried/read by downstream processes
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Flume
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Flume Components Agent: Flume Process. Handles data movement. Source: Handles acquisition of data from Kafka/Hdfs/Http. Pumps it into channel. Sink: Delivers data to destination or another flume agent. Drains the Channel. Channel: A transactional queue backed by memory/disk/other.
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Simple Flume data flow
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Topologies - Aggregation
  13. 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Flume Transactions txnA txnB txnA txnB txnC txnD txnE txnF
  14. 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Flume Fault tolerance and Guarantees à Guarantees: – Depends on the choice of channel – Memory channel : At-least once – If agent does not crash. Data loss of process crashes. – File channel : At-least once – If agent’s disk does not crash. – Kafka channel : At least once – Can tolerate hardware crashes. à Transactional Channel used to provide guarantees.
  15. 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hive Streaming APIs https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest
  16. 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Problems of streaming directly to HDFS à File corruption due to partial writes (due to connection failures) à Need to ensure concurrent writers don t write to same file à Periodic File rotation à Compacting small files into larger à Reading partially written files à Marking files as ‘done’ so that it can be queried/consumed by downstream component
  17. 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hive Streaming APIs à First class support for ingesting data into Hive in real time. à Data is written to HDFS and metadata managed in Hive metastore. APIs manage both. à Transactional in nature. Uses Transaction Batches (i.e. groups of Transactions) à High throughput. à Committed data is readable within a few seconds.
  18. 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hive Streaming APIs - Pseudo Code hiveEP = new HiveEndPoint(metatstoreUrl, db, table, partitions); connection = hiveEP.newConnection(true); writer = new DelimitedInputWriter(fieldNames,",", endPt); txnBatch = connection.fetchTransactionBatch(10, writer); while(txnBatch.remainingTransactions() > 0) { txnBatch.beginNextTransaction(); txnBatch.write("1,Hello streaming".getBytes()); txnBatch.write("2,Welcome to streaming".getBytes()); txnBatch.commit(); }
  19. 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hive Streaming APIs – Guarantees à Guarantees: – At least once
  20. 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Kafka
  21. 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Kafka – Fault tolerance à Brokers : host messages on local disk for readers and writers – Hardware failures or brokers crashing : Messages are replicated among brokers to enable recovery from.
  22. 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Kafka – Data Guarantees à Producers API : Used by client processes to write to Brokers. – When writing, it can choose to • Get an ACK indicating message was committed to all replica brokers • Get an ACK indicating message was committed to lead broker • Not receive any ACK. • Enable idempotent delivery (via configuration) • Use transactional API – Handling write failures: • Enable ACKs and resend – potential for duplication. At Least Once delivery. • (post v0.11.0.0) Enable idempotent delivery. On resend, broker detects and drops any duplicate writes based on sequence numbers assigned to messages by producer and producer IDs. Exactly once delivery to Kafka.
  23. 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Kafka – Fault tolerance & Guarantees à Consumer API : Used by client process to read from Brokers – Consumers periodically commit their current offset to a Kafka topic – Handling Failure: • At least once : If consumer dies, it re-fetches the offset and resumes. At least once semantics possible if you commit the offset after processing the consumed message. • For exactly once: Consumer need a way to store latest consumed offset in the destination along with the outcome of message processing. To recover from failure, load the last offset from destination and start reading from that offset in Kafka.
  24. 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Kafka – Fault tolerance & Guarantees à Read from Kafka --> Write to Kafka – For Exactly Once: Use Transactional API (introduced in 0.11.0.0). Allows writing to multiple topics atomically. Allowing you to save consumed offset (in offsets topic) and outcome of message processing (to destination topic) in the same transaction.
  25. 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Storm
  26. 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Storm – Basics à Distributed Realtime computing engine. à Allows complex processing (joins, aggregations, windowing etc). à Computation performed over set of distributed worker processes. à Worker processes are restarted if they crash.
  27. 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Storm – Data Guarantees à Three modes of usage – Without ACK-ing: At most once guarantee. Message processing is not tracked for proper completion, so no retry attempts in case of failure. Processes on message at a time. – With ACK-ing: At-least once guarantee. Processes on message at a time. Messages are tracked to ensure they are processed by all stages. On error, messages are replayed. Possibility of duplication. – Trident API: Exactly once guarantee if the State being used supports idempotent updates. • Processes messages in batches. The result of processing each batch is committed/updated into an external state store in an idempotent fashion. • Batches are given sequential IDs and their updates will be made in sequential order. • Trident allows multiple in-flight batches and automatically ensures the state updates are made in sequentially order. • Aggregations and Windowing not tied to batch sizes.
  28. 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Flink
  29. 29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Flink – Basics à Distributed Realtime & Batch computing engine. à Computation performed over set of distributed processes. à Job Manager oversees job scheduling, tracking failures and recovery. à Set of Task Managers processes (i.e. workers) execute the job using tasks (threads). à Two Modes of Usage: – Without Checkpoint-ing: At most once guarantee. No tracking of state for recovery on failure. No retry attempts. – With Checkpoint-ing : At-least once OR Exactly once guarantee depending on sinks and sources being used. Stream alignment needs to be enabled for exactly once.
  30. 30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Flink – Checkpointing à Periodically takes a distributed snapshot of the state of all operators. à Latest checkpoint identifies last good state. Checkpoints may be partial or complete (i.e. include state for all participating operators) à On failure: – Picks the latest complete checkpoint – Redeploys entire dataflow, giving each operator its snapshotted state from the chosen checkpoint. – Sources are set to resume reading from the offset associated with the checkpoint
  31. 31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Thank You ! Questions ?

×