Flink's network stack is designed with two goals in mind: (a) having low latency for data passing through, and (b) achieving an optimal throughput. It already achieves a good trade-off between these two but we will continue to tune it further for an optimal one. Flink 1.5 did not only further reduce overhead inside the network stack, but also introduced some groundbreaking changes in the stack itself, introducing an own (credit-based) flow control mechanism and a general change towards a more network-event-driven pull approach. With our own flow control, we steer what data is sent on the wire and make sure that the receiver has capacity to handle it. If there is no capacity, we will backpressure earlier and allow checkpoint barriers to pass through more quickly which improves checkpoint alignment and overall checkpoint speed. In this talk, we will describe in detail how the network stack works with respect to (de)serialization, memory management and buffer pools, the different actor threads involved, configuration, as well as the design to improve throughput and latency trade-offs in the network and processing pipelines. We will go into detail on what is happening during checkpoint alignments and how credit-based flow control improves them. Additionally, we will present common pitfalls as well as debugging techniques to find network-related issues in your Flink jobs.

1. NICO KRUBER
SOLUTION ARCHITECT / SOFTWARE ENGINEER @ DATA ARTISANS,
APACHE FLINK COMMITTER
IMPROVING THROUGHPUT AND LATENCY
WITH FLINK’S NETWORK STACK

2. © 2018 data Artisans2
FLINK DATA TRANSPORT (LOGICAL)
• Subtask output
‒ pipelined-bounded
‒ pipelined-unbounded
‒ Blocking
• Scheduling type
‒ all at once
‒ next stage on complete output
‒ next stage on first output
• Transport
‒ high throughput via buffers
‒ low latency via buffer timeout
Subtask 1
Subtask 2
Subtask 3
Subtask 4
Stream Partition
Abstraction over:

3. © 2018 data Artisans3
TaskManager 1 TaskManager 2
Subtask 4
1
2
Buffer Pool
Subtask 2
3
4
Buffer Pool
Subtask 3
1
2
Buffer Pool
Empty
Buffer
Subtask 1
3
4
Buffer Pool
Buffer with
Data in Queue
FLINK DATA TRANSPORT (PHYSICAL)
TCP Connection

4. © 2018 data Artisans4
TaskManager 1 TaskManager 2
Subtask 2
3
4
Buffer Pool
Subtask 1
3
4
Buffer Pool
Buffer with
Data in Queue
Subtask 4
1
2
Buffer Pool
Subtask 3
1
2
Buffer Pool
Empty
Buffer
FLINK DATA TRANSPORT (PHYSICAL)
TCP Connection
Backpressure

5. © 2018 data Artisans5
TaskManager 1 TaskManager 2
Subtask 2
3
4
Buffer Pool
Subtask 1
3
4
Buffer Pool
Subtask 4
1
2
Buffer Pool
Subtask 3
1
2
Buffer Pool
FLINK DATA TRANSPORT (PHYSICAL)
TCP Connection
Backpressure
Zoom in

6. © 2018 data Artisans6
CREDIT-BASED FLOW CONTROL

7. © 2018 data Artisans7
Subtask 4
TaskManager 1 TaskManager 2
Subtask 2
2
Buffer Pool
CREDIT-BASED FLOW CONTROL (FLINK 1.5+)
TCP Connection
1
Floating
Buffers
Exclusive
Buffers
Backlog

8. © 2018 data Artisans8
Subtask 4
TaskManager 1 TaskManager 2
Subtask 2
2
Buffer Pool
CREDIT-BASED FLOW CONTROL (FLINK 1.5+)
TCP Connection
Floating
Buffers
0
Unannounced
Credit
2
Channel
Credit
announce credit
Send buffers &
announce backlog size
4 Backlog size
Ask for
floating
buffers
321

9. © 2018 data Artisans9
CREDIT-BASED FLOW CONTROL (FLINK 1.5+)
• Never blocks the TCP connection
➢ Better resource utilization with data
skew in multiplexed connections
• Avoids overloading of slow receivers
(direct control over amount of buffered
data)
➢ Improves checkpoint alignment
• cost: additional announce messages
(piggy-bagged),
potential round-trip latency
Checkpoint Duration
Without Flow Control
With Flow Control

10. © 2018 data Artisans10
LOW LATENCY IMPROVEMENTS

11. © 2018 data Artisans11
TaskManager 1 TaskManager 2
Subtask 4
1
2
Subtask 2
Buffer Pool
Subtask 3
1
2
Subtask 1
3
4
NETWORK STACK (EXTENDED)
TCP Connection
NettyServer
Buffer Pool Buffer Pool
Buffer Pool
NettyClient
RecordWriter
3
4
RecordWriter
RecordReader
RecordReader
Zoom in

12. © 2018 data Artisans12
FROM RECORD TO NETWORK
Subtask 1
NettyServer
Buffer
Pool
StreamRecordWriter
Write data &
update writer index
notify
new data
take data &
remove buffer
get new
buffer

13. © 2018 data Artisans13
FROM RECORD TO NETWORK
Subtask 1
NettyServer
Buffer
Pool
StreamRecordWriter
Output
Flusher
flush
notify
new data
Write data &
update writer index
take data &
update reader index

14. © 2018 data Artisans14
BufferConsumer
BufferBuilder
BUFFER BUILDER & CONSUMER
• Producer-Consumer structure with lightweight synchronization
MemorySegment
volatile int
writePosition
int readPosition
• append()
• commit()
• finish() • build() →
Buffer.readOnly
Slice()
Buffer
(wrapping MemorySegment)

15. © 2018 data Artisans15
LATENCY VS. THROUGHPUT
▪ low latency via buffer timeout
*100 nodes x 8 slots
▪ high throughput through buffers

16. © 2018 data Artisans16
CONNECTION TYPES

17. © 2018 data Artisans17
LOCAL VS. REMOTE CONNECTIONS
• Every (unchained) connection:
‒ Requires serialization
‒ Assembles serialized records into buffers
‒ Forwards a buffer when it is full or the buffer timeout hit
• Remote connection:
‒ Sent via multiplexed Netty TCP connections (one per pair of tasks and task managers)
‒ As soon as a buffer is on the wire, it can be re-used
➢ Allows credit-based flow control to control amount of buffered data
• Local connection:
‒ Direct connection between sender and receiver: buffers are shared
➢ No need for further flow control (buffered data = sender buffers)

18. © 2018 data Artisans18
TUNING OPTIONS

19. © 2018 data Artisans19
CREDIT-BASED FLOW CONTROL
• taskmanager.network.credit-model: true/false
• taskmanager.network.memory.buffers-per-channel: 2
• taskmanager.network.memory.floating-buffers-per-gate: 8
• Number of exclusive buffers should be enough to saturate the network for a full
round-trip-time (2 x network latency)
➢ #exclBuffers * segmentSize = round-trip-time * throughput
Subtask 4
TaskManager 1 TaskManager 2
Subtask 2
2
0
Unannounced
Credit2
Channel
Credit
announce credit
Send buffers &
announce backlog size
0
Backlog size

20. © 2018 data Artisans20
CREDIT-BASED FLOW CONTROL
• Number of exclusive buffers too high
➢ higher number of required network buffers
➢ buffering more during checkpoint alignment
➢ BUT: faster ramp-up (before floating buffers kick in)
• Number of exclusive buffers too low
➢ times of in-activity during ramp-up
Subtask 4
TaskManager 1 TaskManager 2
Subtask 2
2
0
Unannounced
Credit2
Channel
Credit
announce credit
Send buffers &
announce backlog size
0
Backlog size

21. © 2018 data Artisans21
BUFFER TIMEOUT
• StreamExecutionEnvironment#setBufferTimeout()
• Affects every unchained connection: remote or local
➢ Upper bound on latency for low throughput channels(!)
➢ Trade-off throughput vs. latency (see earlier)

22. © 2018 data Artisans22
NETWORK THREADS
• netty.client.numThreads (default: number of slots)
• netty.server.numThreads (default: number of slots)
• May become a bottleneck if thread(s) are overloaded
• BUT: may also become an overhead if too many
➢ Do your own benchmarks and verify for your job!

23. © 2018 data Artisans23
USE LINUX-NATIVE EPOLL (FLINK 1.6+)
• taskmanager.network.netty.transport: AUTO | NIO | EPOLL
• EPOLL may reduce the channel polling overhead between user space and
kernel/system space
• There should be no downside in activating this or at least AUTO.
➢ Do your own benchmarks for your job!
• Please give feedback in FLINK-10177 so that we can decide whether to use
AUTO by default.

24. © 2018 data Artisans24
METRICS

25. © 2018 data Artisans25
NETWORK STACK METRICS
• Backpressure monitor
‒ Web/REST UI, /jobs/:jobid/vertices/:vertexid/backpressure)
• [input, output]QueueLength
• numRecords[In, Out]
• numBytesOut, numBytesIn[Local, Remote]
• numBuffersOut, numBuffersIn[Local, Remote] (Flink 1.5.3+, 1.6.1+)

26. © 2018 data Artisans26
LATENCY MARKERS
• ExecutionConfig#setLatencyTrackingInterval() (default: every 2s)
• Sources periodically emit a LatencyMarker with a timestamp
• These flow with the stream and properly queue behind records
• Latency markers bypass operators, e.g. windows
• Once received, they will be re-emitted onto a random output channel
• We create one histogram per source ↔ operator pair (window size: 128)
• source_id.<sourceId>.source_subtask_index.<subtaskIdx>.
operator_id.<operatorId>.operator_subtask_index.<subtaskIdx>
➢ 10 operators, parallelism 100 = 9 * 100 * 100 = 90,000 histograms!
https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#latency-tracking

27. © 2018 data Artisans27
COMMON ANTIPATTERNS

28. © 2018 data Artisans28
REPEATED KEYBY’S ON THE SAME KEY
• KeyedStream is not retained
‒ UDF could have changed the key
• Additional keyBy() is necessary to gain access to keyed state, but:
‒ Prevents chaining
‒ Adds an additional shuffle
➢ DataStreamUtils#reinterpretAsKeyedStream
.keyBy(“location”)
.keyBy(“location”)

29. © 2018 data Artisans29
Subtask 4
TaskManager 1 TaskManager 2
Subtask 2
Buffer Pool
CREDIT-BASED FLOW CONTROL (FLINK 1.5+)
TCP Connection
Floating
Buffers
Exclusive
Buffers
Buffers: #channels * 2 + 8
LatencyMarker
➢ synchronization overhead
for the output flusher!

30. © 2018 data Artisans30
WHAT‘S UP NEXT?

31. © 2018 data Artisans31
NETWORK SERIALIZATION STACK (FLINK 1.7?)
• Serialization for broadcasts once per record, not channel
• Only one intermediate serialization buffer (on heap)
➢ significantly reduces the memory footprint
• see FLINK-9913
TaskManager 1
Subtask 2
RecordWriter

32. © 2018 data Artisans32
OPENSSL-BASED SSL ENGINE (FLINK 1.7?)
• Runs native code
• Uses advanced CPU instruction sets
➢ May reduce encryption/decryption overhead (needs verification)
• see FLINK-9816

33. © 2018 data Artisans33
MOVE OUTPUT FLUSHER TO NETTY
• Current implementation may have (GC) problems with many channels
➢ schedule the output flusher inside the Netty event loop
• see FLINK-8625

34. THANK YOU!
@dataArtisans
@ApacheFlink
WE ARE HIRING
data-artisans.com/careers