This document discusses exactly once and effectively once semantics in Heron stream processing. It explains that exactly once is not truly possible and effectively once uses an optimistic approach with distributed snapshots and markers to provide exactly once semantics. The document compares optimistic and pessimistic approaches, and describes how Heron implements the optimistic approach using state managers and a checkpoint manager with pluggable state stores. It also discusses challenges with bolt state, sink state, and ensuring idempotent or transactional writes.

1. Exactly Once, Effectively Once
in Heron
Karthik Ramasamy, Sanjeev Kulkarni

2. 2
Exactly Once Semantics
What is it?
Data
Need to be read/wri>en exactly once
Compute
Tuple need to be processed exactly once

3. 3
Exactly Once Semantics
Is it really possible?
Data TransacEonality
Do (publish tuple, record tuple) atomically
Do (consume tuple, record tuple) atomically
Only possible by having some kind of idempotent (eg. sequence ids)
Compute TransacEonality
Do (execute tuple, state change) atomically
May not be possible if state involves external enEEes (like update a db)
Only possible by having idempotent operaEon where changed state is abstracted by system

4. 4
Exactly Once Semantics
Is really
Effectively Once

5. 5
Effectively Once Requirements
AddiEon of effecEvely once should not impact the performance and funcEonality
of at least once and at most once topologies in producEon
Resource consumpEon for exisEng topologies should not change when effecEvely
once is supported
State store to support effecEvely once should provide an abstracEon for different
types of storage
Should support topologies of various sizes

6. 6
Effectively Once Semantics
OpEmisEc
Go as fast as you can and assume failures are rare
PessimisEc
Assumes failures are the norm

7. 7
Effectively Once Semantics
OpEmisEc Approach
Distributed Snapshots
Markers flowing from sources all the way to downstream components
Nodes checkpoint when they see the markers
On failure, rollback to a globally consistent state
Globally consistent state - all components completed their checkpoint

8. 8
Effectively Once Semantics
OpEmisEc Approach
S
B1
B2
B3

9. 9
Effectively Once Semantics
OpEmisEc Approach
S
B1
B2
B3

10. 10
Effectively Once Semantics
OpEmisEc Approach
S
B1
B2
B3
M
M

11. 11
Effectively Once Semantics
OpEmisEc Approach
S
B1
B2
B3
M
M

12. 12
Effectively Once Semantics
OpEmisEc Approach
S
B1
B2
B3
M
M
M
M

13. 13
Effectively Once Semantics
OpEmisEc Approach
S
B1
B2
B3
M
M
M
M

14. 14
Effectively Once Semantics
OpEmisEc Approach
S
B1
B2
B3
M
M
M
M
Globally consistent State

15. 15
Effectively Once Semantics
Advantages of OpEmisEc Approach
Minimal compute overhead
Marker tuples once in a set interval
State snapshoYng happens in parallel
No central locking to get distributed state
Minimal storage overhead
State per processing node per interval
Can be cleaned up to keep only the last few ones

16. 16
Effectively Once Semantics
Disadvantages of OpEmisEc Approach
Recovery is global
Revert topology for any component failure
All components stop processing and revert to a previous state
Scale issues
The bigger the topology, the probability of failure increases
In worst case, topology may not make progress

17. 17
Effectively Once Semantics
PessimisEc Approach
At each processing stage
Record all incoming tuples, unEl they are processed
Record all outgoing tuples aZer they are produced
Maintain watermarks between stages

18. 18
Effectively Once Semantics
PessimisEc Approach
Input Tuples Output Tuples

19. 19
Effectively Once Semantics
PessimisEc Approach
Input Tuples Output Tuples
Storage

20. 20
Effectively Once Semantics
Advantages of PessimisEc Approach
Recovery is local
Component failure results in recovery of that component
Other components are not affected
Scales in depth and size
Bigger topology doesn’t affect topology progress

21. 21
Effectively Once Semantics
Disadvantages of PessimisEc Approach
Latency will be very high
Resource Intensive
Need to save all intermediate streams
Resources needed might exceed by several magnitudes
Need a high throughput and durable storage

22. 22
Effectively Once Semantics for Heron
Decision points
Cost was the biggest factor
Twi>er had very high thrust topologies
Topologies that needed exactly once had small logical plan
Favored OpEmisEc

23. 23
Effectively Once Semantics in Heron
Uses OpEmisEc Approach
High throughput
Low latency
Low cost
Pluggable state storage
Hadoop, LFS, BookKeeper
Easy to enable
Available in Java and Python

24. 24
Effectively Once Semantics for Heron
Involvement of Stmgr
State flows from instance to the stmgr to checkpointmgr
Instances deal only with stmgr
Adding addiEonal languages becomes trivial
Asynchronous operaEon

25. 25
Effectively Once Semantics for Heron
Checkpoint Manager
Isolates State Store interacEon
Running in all containers
Configurable State Stores
Supports HDFS/S3/Bookkeeper State stores

26. 26
Effectively Once Semantics
State ExaminaEon
Spouts
Offset
Bolts
Any internal state
Sinks
Writes aZer marker?

27. 27
Effectively Once Semantics
Bolt State
Large State
Inefficient to store it for every checkpoint interval
Pressure on State Store
Network bandwidth
Possible way outs
Incremental State

28. 28
Effectively Once Semantics
Incremental state
State RepresentaEon
Actual State
Update Set since last checkpoint marker
State Saving
Update Set + Reference to previous state
CompacEon of Update Set
Periodically send whole state

29. 29
Effectively Once Semantics
Sink State Problem
State Rewind
What about writes since last checkpoint
Duplicate writes
PotenEal SoluEons
TransacEonal writes
Idempotent writes

30. 30
Effectively Once Semantics
TransacEonal Writes
Pre-Commit Stage
Writes are wri>en and staged, but not yet commi>ed
Thrown away if aborted
Commit Stage
Writes become visible to the world
Cannot be taken back

31. 31
Effectively Once Semantics
Advantages of TransacEonal Writes
Straigheorward interface
Easy implementaEons for file system/database sinks

32. 32
Effectively Once Semantics
Disadvantages of TransacEonal Writes
Need support from Sinks
Commit Stage
Degenerates to Micro-Batch mode
Increased Write bandwidth in failures

33. 33
Effectively Once Semantics
Idempotent Writes aka EffecEvely Once Writes
Sequence Id per write
Monotonically increasing Sequence Id
Write side de-duplicaEon
Ignore Duplicate writes

34. 34
Effectively Once Semantics
Advantages of EffecEvely Once writes
Simple interface
Simple implementaEons for File and Message Queue Sinks
Write bandwidth unaffected by failures
Sinks just ignore duplicate writes
No extra Commit Stage

35. 35
Effectively Once Semantics
Disadvantages of EffecEvely Once writes
Need support from Sinks
Progress mismatch
World might see output while duplicate computaEon sEll ongoing

37. 37
More to Come…
Apache
Pulsar
Apache
Pulsar
Apache BookKeeper
Heron