Apache Kafka Performance with Dell EMC Isilon F800 All-Flash NAS

Abstract
This paper describes performance test results for running Kafka with Dell
EMC Isilon F800 All-Flash NAS Storage. A comparison against direct
attached storage is also provided.
20+ MILLION RECORDS A SECOND
Running Kafka with Dell EMC Isilon All Flash F800 Scale-out NAS
Author: Boni Bruno, CISSP, CISM, CGEIT
Chief Solutions Architect, Dell EMC
Authotr

Apache Kafka Performance with Dell EMC Isilon F800 All-Flash NAS
Copyright © August 2018 Dell Inc. or its subsidiaries. All rights reserved.
Dell believes the information in this publication is accurate as of its publication date. The
information is subject to change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS-IS.“ DELL MAKES NO REPRESENTATIONS
OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND
SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN
THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.
Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks
may be the property of their respective owners.
EMC Corporation
Hopkinton, Massachusetts 01748-9103
1-508-435-1000 In North America 1-866-464-7381
www.EMC.com

Contents
Overview.......................................................................................................................................................................... 4
Kafka Introduction ......................................................................................................................................................... 5
Dell EMC Isilon F800 All-Flash NAS................................................................................................................................ 8
Performance Test Environment.................................................................................................................................. 11
Performance Test Results ............................................................................................................................................ 12
Test 1 - Single producer (1 Broker), no replication, 50M records, 100 Byte record size ........................... 12
Test 2 - Single producer (5 brokers), 3x asynchronous replication, 50M records, 100 Byte record size 13
Test 3 - Single producer (5 brokers), 3x synchronous replication, 50M records, 100 Byte record size .. 15
Test 4 - 5 producers, no replication, 250M records, 100 Byte record size .................................................. 16
Test 5 - 5 producers, 3x asynchronous replication, 250M records, 100 Byte record size......................... 17
Test 6 – Effect of Record Size on Producer Throughput ................................................................................ 19
Test 7 - Single consumer, 50M records, 100 Byte record size........................................................................ 21
Test 8 - 5 consumers, 250M records, 100 Byte record size............................................................................. 22
Test 9 – 1 producer & 1 consumers, 50M records written, 50M records read ........................................... 23
Test 10 – Stress testing Isilon F800 All-Flash Scale-out NAS ............................................................................. 24
Conclusions ................................................................................................................................................................... 27
Appendix ....................................................................................................................................................................... 28
Kafka Server Properties........................................................................................................................................ 29
Zookeeper Properties........................................................................................................................................... 30
Producer Properties.............................................................................................................................................. 30
Consumer Properties............................................................................................................................................ 30
NFS Client Configuration ..................................................................................................................................... 31
Isilon Configuration............................................................................................................................................... 33
OneFS TCP Tuning................................................................................................................................................. 37
Kafka End-to-End Latency Test .......................................................................................................................... 39

Overview
Kafka is a distributed, horizontally-scalable, fault-tolerant, stream processing system being used in many
enterprises. Kafka is a system that lets you publish and subscribe to streams of data, it also stores and
process the data. It is now a part of the Apache Software Foundation with a commercial version
available through Confluent that includes Kafka software enhancements and Enterprise level of
support.
Kafka runs as a cluster and can scale to handle millions of records a second. This paper covers Kafka
performance test results for a Kafka DAS (Direct Attached Storage) cluster using PowerEdge R730XD
servers and a Kafka Isilon F800 NAS (Network Attached Storage) cluster using the same servers.
As you read through this paper, you will see that Dell EMC Isilon F800 Scale-out NAS solution provides
excellent performance and better storage utilization with less drives and a smaller storage foot print
compared to DAS. This is great news for customers looking to simplify their Kafka cluster deployments
and improve storage efficiency.

Kafka Introduction
A Kafka cluster consists of Producers that send records to the cluster, the cluster stores these records and
makes them available to Consumers. A general Kafka cluster diagram is shown below for reference.
A key concept to understand with Kafka is what is known as a Topic. Producers publish their records to
a specific topic and consumers can subscribe to one or more of these topics. A Kafka topic is just a
partitioned write-ahead log. Producers append records to these logs and consumers simply subscribe
to the changes. The records consist of a key/value pair. The key is used for assigning the records to a
log partition. Below is an example of a topic with four partitions with writes being appended to the end
of each partition.
Partitions also provide redundancy and scalability. Each partition can be hosted on a different server
allowing a single topic to be scaled horizontally to increases cluster performance. The term stream in
Kafka is a single topic of data regardless of the number of partitions.
Consumers work as part of a consumer group where one or more consumers work together to consume
a topic. Each partition is only consumed by one member of the consumer group. Below is an example

of three consumers in a single group consuming a single topic. Here two consumers are working from
one partition and the third consumer is working from two partitions.
Consumers can scale to consume topics with large number of messages. If a single consumer fails, the
remaining members of the group will rebalance the partitions being consumed to take over for the
failed consumer.
A single Kafka server if called a Broker. Brokers receive messages from producers, assigns offsets to
them, and commits the messages to disk. Brokers also service requests from consumers. Brokers are part
of the Kafka Cluster, only one broker will be elected as the cluster controller to assign partitions to
brokers and detect broker failures. A partition is owned by a single broker, this broker becomes the
leader of the partition. If a partition is assigned to multiple brokers, the partition will be replicated to
provide better redundancy if a broker were to fail.
The diagram below shows replication of partitions in Kafka.

Kafka allows retention policies to be configured where Kafka Brokers retain messages for some period of
time or until a topic reaches a certain size in bytes. Once these limits are reached, messages are
expired and deleted.
Storage management gets complicated with Kafka as retention requirements increase or as the Kafka
cluster itself increases in size. Centralizing storage with Dell EMC Isilon can greatly simplify storage
management issues with Kafka as more space is needed, with Isilon you simply add more Isilon nodes to
the backend and capacity is instantly available with no need to change any of the Kafka configuration
except for maybe increasing the retention policy.
The next section covers Dell EMC Isilon in more detail.

Dell EMC Isilon F800 All-Flash NAS
Dell EMC Isilon F800 all-flash scale-out NAS storage provides up to 250,000 IOPS and 15 GB/s bandwidth
per chassis. With a choice of SSD drive capacities, all-flash storage ranges from 96 TB to 924 TB per
chassis making the Isilon F800 ideal for demanding storage requirements in high volume messaging
systems like Kafka.
In additional to all-flash high-performance scale-out hardware design of the Isilon F800, the embedded
storage operating system (Isilon OneFS) provides a unifying clustered file system with built-in scalable
data protection that simplifies storage management and administration. OneFS is a fully symmetric file
system with no single point of failure — taking advantage of clustering not just to scale performance
and capacity, but also to allow for any-to-any failover and multiple levels of redundancy that go far
beyond the capabilities of RAID.
OneFS allows hardware to be incorporated or removed from the cluster at will and at any time,
abstracting the data and applications away from the hardware. Data is given infinite longevity and the
cost and pain of data migrations and hardware refreshes are eliminated.
Isilon nodes
OneFS works exclusively with the Isilon scale-out NAS nodes, referred to as a “cluster”. A single Isilon
cluster consists of multiple nodes, which are rack-mountable enterprise appliances containing: memory,
CPU, networking, Ethernet or low-latency InfiniBand interconnects, disk controllers and storage media.
As such, each node in the distributed cluster has compute as well as storage capabilities.
With the new generation of Isilon hardware (“Gen6”), a single chassis of 4 nodes in a 4U form factor is
required to create a cluster, which currently scales up to 144-nodes. Previous Isilon hardware platforms
need a minimum of three nodes and 6U of rack space to form a cluster. There are several different
types of nodes, all of which can be incorporated into a single cluster, where different nodes provide
varying ratios of capacity to throughput or Input/Output operations per second (IOPS).
Each node or chassis added to a cluster increases aggregate disk, cache, CPU, and network capacity.
OneFS leverages each of the hardware building blocks, so that the whole becomes greater than the
sum of the parts. The RAM is grouped together into a single coherent cache, allowing I/O on any part of
the cluster to benefit from data cached anywhere. A file system journal ensures that writes are safe
across power failures. Spindles and CPU are combined to increase throughput, capacity and IOPS as
the cluster grows, for access to one file or for multiple files. A cluster’s storage capacity can range from
a minimum of 18 terabytes (TB) to a maximum of greater than 68 petabytes (PB). The maximum
capacity will continue to increase as disk drives and node chassis continue to get denser.
Isilon nodes are broken into several classes, or tiers, according to their functionality:

This paper focuses on the F800 node type for Kafka. A good alternative to the F800 is the H600 node
type if storage capacity requirements are lower.
Network
There are two types of networks associated with a cluster: internal and external.
Back-end (internal) network
All intra-node communication in a cluster is performed across a dedicated backend network,
comprising either 10 or 40 GbE Ethernet, or low-latency QDR InfiniBand (IB). This back-end network,
which is configured with redundant switches for high availability, acts as the backplane for the cluster.
This enables each node to act as a contributor in the cluster and isolating node-to-node
communication to a private, high-speed, low-latency network. This back-end network utilizes Internet
Protocol (IP) for node-to-node communication.
Front-end (external) network
Clients connect to the cluster using Ethernet connections (1GbE, 10GbE or 40GbE) that are available on
all nodes. Because each node provides its own Ethernet ports, the amount of network bandwidth
available to the cluster scales linearly with performance and capacity. The Isilon cluster supports
standard network communication protocols to a customer network, including NFS, SMB, HTTP, FTP, HDFS,
and OpenStack Swift. Additionally, OneFS provides full integration with both IPv4 and IPv6 environments.
The Kafka Isilon F800 cluster tested in this paper uses NFS v3 as the network communication protocol.
Complete cluster view

The complete cluster is combined with hardware, software, networks in the following view:
File system structure
The OneFS file system is based on the UNIX file system (UFS) and, hence, is a very fast distributed file
system. Each cluster creates a single namespace and file system. This means that the file system is
distributed across all nodes in the cluster and is accessible by clients connecting to any node in the
cluster. There is no partitioning, and no need for volume creation.
Because all information is shared among nodes across the internal network, data can be written to or
read from any node, thus optimizing performance when multiple users or applications are concurrently
reading and writing to the same set of data.
For more details on Isilon and OneFS please see Isilon Technical Overview.

Performance Test Environment
Kafka
The Kafka version tested for this paper is version 2.12-1.1.0. Kafka configuration (including zookeeper)
properties are shown in the Appendix.
Compute Nodes
All the compute nodes are identical Dell PowerEdge R730xd servers with 40 cores, 256G RAM, 25 x 1.1 TB
SAS disks (directly mounted JBOD, no RAID), and 10G NIC running CentOS Linux release 7.4.1708 (Core).
Up to 12 x PowerEdge R730xd servers were used for various test scenarios that are described in detail in
the Performance Test Results section of this paper. A total of 5 zookeeper servers are configured in the
test environment and run on the first five Kafka servers – k0, k1, k2, k3, and k4. The remaining Kafka
servers are named k5 – k11.
Isilon F800 (NFS Mounted from each Kafka compute node)
A single Isilon F800 Chassis with 60 x 1.6TB SSD drives available for Kafka-Isilon testing.
The specific Isilon Model tested: Isilon F800-4U-Single-256GB-1x1GE-2x40GE SFP+-24TB SSD
The Isilon OneFS release tested: OneFS v 8.1.0.4. NFS configuration details are listed in the Appendix.
Kafka Clusters
Two Kafka clusters were tested - a DAS cluster with 300 x SAS drives vs a single 4U Isilon F800 cluster with
only 60 drives.
The DAS cluster strictly uses PowerEdge R730xd servers for both compute and storage, a total of 12
PowerEdge servers with 300 SAS disks were available (~ 300 TB capacity) for the DAS Kafka cluster. All
the compute nodes were connected 10GbE with jumbo frames (MTU 9014) enable.
The Isilon cluster uses a single 4U Isilon F800 for all Kafka storage with the PowerEdge R730xd servers used
for compute only. Each Kafka server NFS mounts a corresponding kafka-logs directory from Isilon.
Each server has a unique mount point. Details on the NFS setup is shown in the Appendix. Only the OS
drive and an additional 1.1 TB drive was used on each PowerEdge server for the Isilon cluster. The Isilon
F800 connected to the 10GigE PowerEdge environment over 40GbE ports on the core switch with jumbo
frames (MTU 9000) enabled.
Note: Only half the 40GbE front-end ports available on Isilon were connected due to the lack of
available 40GbE ports on the core switch during testing.

Performance Test Results
Producer Throughput Tests
The producer throughput tests stress the throughput of the producer on each cluster (DAS & Isilon). No
consumers are run during these tests so all messages are persisted but not read. Results below show the
average of three test runs.
Note: The optimum batch size is used for each cluster, this is determined by various test runs for each
cluster and seeing what yields the best performance result.
Test 1 - Single producer (1 Broker), no replication, 50M records, 100 Byte record size
Producer Test 1 Setup:
DAS:
kafka-topics.sh –-zookeeper k0:2181 –-create –-topic DASr1 –-partitions 8 –-replication factor 1
ISILON:
kafka-topics.sh --zookeeper k0:2181 --create --topic F800r1 --partitions 8 --replication-factor 1
Producer Test 1 Commands:
DAS:
kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --topic DASr1 --num-records 50000000 --
throughput -1 --record-size 100 --producer-props acks=1 bootstrap.servers=k0:9092 buffer.memory=67108864
batch.size=8196
ISILON:
bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --topic rep1 --num-records 50000000 --
batch.size=194196
DAS Producer Throughput Result: 50,000,000 records sent, 1,231,861 records/sec (117 MB/sec)
25ms average latency, 42ms 95th percentile latency
ISILON Producer Throughput Result: 50,000,000 records sent, 1,401,424 records/sec (134 MB/sec)

Test 2 - Single producer (5 brokers), 3x asynchronous replication, 50M records, 100 Byte record size
Test 2 is exactly the same as the previous one except that now each partition has three replicas (so the
total data written to cluster is three times greater). Each server is doing both writes from the producer for
the partitions for which it is a master, as well as fetching and writing data for the partitions for which it is
a follower.
Replication in this test is asynchronous. That is, the server acknowledges the write as soon as it has
written it to its local log without waiting for the other replicas to also acknowledge it. This means, if the
master were to crash, it would likely lose the last few messages that had been written but not yet
replicated. This makes the message acknowledgement latency a little better at the cost of some risk in
the case of server failure.
When using a JBOD configuration on DAS, replication is important to increase redundancy, however the
total cluster write capacity is 3x less with 3x replication (since each write is done three times). Isilon uses
erasure coding to increase redundancy and improve storage efficiency (near 80% efficiency). Also, the
Isilon high-speed interconnect provides data accessibility across all nodes, even when there is a node
failure on Isilon, data can be retrieved from the remaining nodes without any re-elections on Kafka.
DAS:
kafka-topics.sh –-zookeeper k0:2181,k1:2181,k2:2181,k3:2181,k4:2181 –-create –-topic DASr3 –-partitions
8 –-replication factor 3
ISILON:
kafka-topics.sh --zookeeper k0:2181,k1:2181,k2:2181,k3:2181,k4:2181 --create --topic F800r3 --partitions
8 --replication-factor 3

DAS:
batch.size=8196
ISILON:
kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --topic F800r3 --num-records 50000000 --
batch.size=194196
Asynchronous replication (acks=1) does decrease throughput, increases latency, and uses up more
storage space. When using Isilon, 3x data replication is not needed based on the redundancy and
efficiency built into Isilon/OneFS, but it is recommended for DAS Kafka cluster deployments.
For Isilon, a replication of 1 is fine and will offer the best throughput and storage efficiency, use a
replication of 2 to provide better Kafka server fault tolerance if Kafka compute node failure is a concern
when deploying Kafka with Isilon.

Test 3 - Single producer (5 brokers), 3x synchronous replication, 50M records, 100 Byte record size
Test 3 is the same as Test 2 except that now the master for a partition waits for acknowledgement from
the full set of in-sync replicas before acknowledging back to the producer. With synchronous
replication, Kafka ensures that messages will not be lost as long as one in-sync replica remains.
Synchronous replication (acks = -1) in Kafka is not fundamentally very different from asynchronous
replication. The leader for a partition always tracks the progress of the follower replicas, Kafka will not
send out messages to consumers until they are fully acknowledged by replicas. With synchronous
replication Kafka waits to respond to the producer request until the followers have replicated it.
DAS:
throughput -1 --record-size 100 --producer-props acks=-1 bootstrap.servers=k0:9092
buffer.memory=67108864 batch.size=8196
ISILON:
kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --topic F800r3 --num-records 50000000 --
throughput -1 --record-size 100 --producer-props acks=-1 bootstrap.servers=k0:9092
DAS Producer Throughput Result: 50,000,000 records sent, 269,879 records/sec (26 MB/sec)
ISILON Producer Throughput Result: 50,000,000 records sent, 1,046,703records/sec (100 MB/sec)
Synchronous replication (acks = -1) does decrease throughput significantly and also introduces
significant latency on the DAS cluster. In all three single producer test cases, the throughput and
latency results were better with the Isilon NAS cluster even though Isilon had less disks.

Test 4 - 5 producers, no replication, 250M records, 100 Byte record size
Test 4 is the same as Test1 except now we have increase the number of producers to 5 and the
generated record load to 250M records for each DAS and Isilon NAS Kafka cluster.
DAS:
kafka-topics.sh –-zookeeper k0:2181,k1:2181,k2:2181,k3:2181,k4:2181 –-create –-topic 5DASr1 –-partitions
ISILON:
kafka-topics.sh --zookeeper k0:2181,k1:2181,k2:2181,k3:2181,k4:2181 --create --topic 5F800r1 --
partitions 8 --replication-factor 1
Note: The commands are run simultaneously on each of the 5 Kafka producer servers (k0-k5), the only
change to each command is with the bootstrap.servers parameter, the local producer is referenced
with each command, i.e. server k0 used bootstrap.servers k0, k1 uses k1, etc. This has nothing to do with
where partitions are written to, this just tells Kafka where to pull the bootstrap information from.
DAS:
kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --topic 5DASr1 --num-records 50000000 --
batch.size=8196
ISILON:
kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --topic 5F800r1 --num-records 50000000 --
batch.size=194196
Note: The records/sec and MB/sec results shown represent the aggregate sum across the 5 producers.
The latency results shown are from the slowest producer in the.

Test 5 - 5 producers, 3x asynchronous replication, 250M records, 100 Byte record size
Test 5 is the same as Test4 except now 3x asynchronous data replication is configured. This is not a
needed (definitely not recommended) configuration for Isilon. This configuration only makes sense
when using a Kafka DAS cluster. Isilon provides much better data redundancy than DAS, however Isilon
does not provide protection against a Kafka broker failure. Use 2x asynchronous replication to protect
against Kafka broker failures when using Kafka with Isilon, otherwise stick to a replication factor of 1
when using Isilon with Kafka to ensure optimum performance and storage efficiency.
DAS:
kafka-topics.sh –-zookeeper k0:2181,k1:2181,k2:2181,k3:2181,k4:2181 –-create –-topic 5DASr3 –-partitions
ISILON:
kafka-topics.sh --zookeeper k0:2181,k1:2181,k2:2181,k3:2181,k4:2181 --create --topic 5F800r3 --
DAS:
kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --topic 5DASr3 --num-records 50000000 --
batch.size=8196
ISILON:
kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --topic 5F800r5 --num-records 50000000 --
batch.size=194196
186ms average latency, 1,081ms 95th percentile latency

As expected, 3x asynchronous replication increases latency on both clusters. A 3x replication factor
configuration with Isilon will put both the original data and replicated data all on Isilon since each Kafka
server is NFS mounting the associated kafka-logs directory from Isilon (see NFS configuration in Appendix
for details). This configuration unnecessarily adds a lot of network traffic and I/O requests to the Isilon
cluster which should be avoided. Alternatively, the DAS cluster only replicates data to a select number
of servers. Isilon does not need this amount of replication to provide data redundancy. Stick to
replication factor of 1 or at most a replication factor of 2 when using Isilon NAS with Kafka.

Test 6 – Effect of Record Size on Producer Throughput
So far all producer tests have used a small record size of 100 Bytes, this is not representative of most
production deployments of Kafka, but is a good size to use for stress testing purposes.
To get a better idea of individual producer throughput performance, different record sizes from 10 Bytes
to 100,000 Bytes were tested to see the effect on producer throughput.
Note: Producer throughput can be measured in two ways – number of records processed per second
or the byte throughput per second (MB/sec). Both results are shown below for reference.
1516203
1281251
217367
30166
3272
1585389
1294378
180323
32912
3836
0
200000
400000
600000
800000
1000000
1200000
1400000
1600000
1800000
10 100 1000 10000 100000
Records/sec
Record Size (Bytes)
Impact of Record Size on Throughput (Records/sec)
DAS
F800

The graphs above show that the number of records Kafka can send per second decreases as the
records get larger is size. If we look at MB/second results, the total byte throughput increases as
messages get bigger. It is important to understand the typical record size in your environment so you
can size your Kafka cluster accordingly to meet your throughput requirements.
The results shown above are specific to using PowerEdge R730XD servers with the stated specifications,
different compute specifications will have different throughput results. Test your specific compute
model to get a good understanding of the Kafka throughput capabilities for your particular compute
nodes.
14
122
207
288 312
15
123
172
314
366
0
50
100
150
200
250
300
350
400
10 100 1000 10000 100000
MB/sec
Record Size (Bytes)
Impact of Record Size on Throughput (MB/sec)
DAS F800

Consumer Throughput Tests
So far only producer testing has been conducted, producer tests provide insights on the write
capabilities of Kafka. As shown in the Kafka producer tests above, both the DAS and Isilon clusters can
write over 5M records a second with only 5 producers.
A key value proposition to note so far is that Isilon is providing very good performance with less spindles,
less power, and a smaller storage foot print. The 5 producers tested ran on 2U PowerEdge R730xd
servers with 25 drives each, thus the 5 node producer DAS Kafka cluster tested has a total of 125 disks
and takes 10U of rack space. Adhering to Kafka best practices for DAS clusters and using a replication
factor of 3, the approximate useable storage for the DAS cluster is only 42 TB (125 TB / 3).
On the other hand, Isilon offers similar performance with half the number of disk drives and uses only 4U
of rack space and provides over 80 TB of usable storage! This is great news for Kafka administrators and
organizations looking to reduce the foot print of their Kafka clusters without sacrificing performance.
Now let’s look at the read capabilities of Kafka by running various consumer throughput tests against
both DAS and Isilon NAS Kafka clusters. Note that the replication factor will not affect the outcome of
the consumer tests as the consumer only reads from one replica regardless of the replication factor.
Likewise, the acknowledgement level of the producer also doesn't matter as the consumer only ever
reads fully acknowledged messages.
Test 7 - Single consumer, 50M records, 100 Byte record size
Consumer Test 7 commands:
DAS:
kafka-consumer-perf-test.sh --zookeeper k0:2181 --messages 50000000 --topic DASr3 --threads 1
ISILON:
kafka-consumer-perf-test.sh --zookeeper k0:2181 --messages 50000000 --topic F800r3 --threads 1
DAS Consumer Throughput Result: 50,000,000 records read, 1,763,544 records/sec (168 MB/sec)
ISILON Consumer Throughput Result: 50,000,000 records read, 1,795,139 records/sec (171 MB/sec)

Test 8 - 5 consumers, 250M records, 100 Byte record size
Consumer Test 8 commands: (Executed on 5 Kafka consumers simultaneously)
DAS:
kafka-consumer-perf-test.sh --zookeeper k0:2181 --messages 50000000 --topic 5DASr3 --threads 1
ISILON:
kafka-consumer-perf-test.sh --zookeeper k0:2181 --messages 50000000 --topic 5F800r3 --threads 1

Test 9 – 1 producer & 1 consumers, 50M records written, 50M records read
The performance tests conducted thus far covered just the Kafka producers and the Kafka consumers
running in isolation. A typical deployment of Kafka runs the producer and consumer together.
Technically, the performance tests above have been running both the producer and consumers
together as Kafka replication works by using the servers themselves as consumers.
For this test one producer and one consumer is run against an eight partition 3x replicated topic that
begins empty. The producer is using async replication. The throughput reported is the consumer
throughput.
Test 9 setup commands:
DAS:
kafka-topics.sh --zookeeper=k0:2181,k1:2181,k2:2181,k3:2181,k4:2181 --create --partitions 8 --
replication-factor 3 --topic r3DASnew
ISILON:
kafka-topics.sh --zookeeper=k0:2181,k1:2181,k2:2181,k3:2181,k4:2181 --create --partitions 8 --
replication-factor 3 --topic r3F800new
Test 9 test commands:
DAS Producer:
kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --topic r3DASnew --num-records 50000000 --
batch.size=8196
DAS Consumer:
kafka-consumer-perf-test.sh --zookeeper k0:2181 --messages 50000000 --topic r3DASnew --threads 1
ISILON Producer:
kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --topic r3F800new --num-records 50000000 -
-throughput -1 --record-size 100 --producer-props acks=1 bootstrap.servers=k1:9092
ISILON Consumer:
kafka-consumer-perf-test.sh --zookeeper k0:2181 --messages 50000000 --topic r3F800new --threads 1

Test 10 – Stress testing Isilon F800 All-Flash Scale-out NAS
A single Isilon F800, as tested in this paper, only comes with 4 nodes. Isilon normally recommends having
one Isilon node for each compute node in a high performance distributed cluster. The previous results
already show that the Isilon F800 can easily handle a 5 node Kafka cluster.
This test will increase the compute count to 12 nodes to see how well a 4-node F800 Isilon cluster can
support 12 Kafka servers simultaneously generating 50M records then consuming 50M records from the
same Kafka Topic. This equates to generating and consuming 600M records in total with just a single
Isilon F800 chassis that has 4 nodes and 60 drives. Furthermore, three different record sizes will be tested,
10 Bytes, 100 Bytes, and 512 Bytes.
Test 10 Setup:
kafka-topics.sh --create --zookeeper=k0:2181,k1:2181,k2:2181,k3:2181,k4:2181 --topic 12nodes10 --
Test 10 commands: (Executed on 12 Kafka server nodes simultaneously)
Producer commands:
10 Byte Test:
kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --topic 12nodes100 --num-records 50000000
--throughput -1 --record-size 10 --producer-props acks=1 bootstrap.servers=k0:9092
100 Byte Test:
512 Byte Test:
Consumer commands:
10 Byte Test:
kafka-consumer-perf-test.sh --zookeeper k0:2181 --messages 50000000 --topic 12nodes10 --threads 1
100 Byte Test:
512 Byte Test:

Test 10 Results:
Producer results:
ISILON 10B Throughput Result: 600,000,000 records, 17,719,448 records/sec (168 MB/sec)
ISILON 100B Throughput Result: 600,000,000 records, 12,161,606 records/sec (1,158 MB/sec)
ISILON 512B Throughput Result: 600,000,000 records, 4,335,062 records/sec (2,115 MB/sec)
Consumer results:
ISILON 10B Throughput Result: 600,000,000 records read, 19,141,011 records/sec (1,825 MB/sec)
A single Isilon F800 performed very well under high load. The Kafka results show that producers can write
over 17+ million records a second and consumers can read over 20+ million records a second with a
single F800 Isilon chassis.
All the performance results shown thus far are directly from Kafka. We can also get stats directly from
Isilon while under load to see the utilization of the individual Isilon nodes, network throughput rates, disk
throughput rates, cpu utilization, active number of clients, etc.
Below are Isilon specific performance reporting charts during the 10 Byte, 100 Byte, and 512 Byte tests
with 12 Kafka servers.

Isilon Performance Report during 10 Byte Record Size Test
Isilon Performance Report during 100 Byte Record Size

Isilon Performance Report during 512 Byte Record Size
You can see from the Isilon performance reports above the all four nodes show even load distribution
for network and disk throughput rates. Load balancing network connections and disk I/O is a key value
proposition Isilon provides and it’s automatic. All the Kafka nodes just NFS mount Isilon by its FQDN (fully
qualified domain name) and OneFS transparently handles all the load balancing for you.
Also note that the CPU utilization stayed low during all test runs. This means the F800 has resources to
spare while the stress test was running, this is a testament to the engineering work that went into the
product.
Conclusions
This paper shows that a single Isilon F800 performed very well under high load with Kafka. The Kafka
results show that producers can write over 17+ million records a second and consumers can read over
20+ million records a second using a single Isilon F800 NAS system. During all performance tests, the
CPU utilization across the entire Isilon cluster stayed very low.
The Isilon F800 Scale-out NAS storage system performed just as well as a Kafka DAS (direct attached
storage) cluster that had 5x the number of disks. The embedded erasure code design with Isilon OneFS
also provides much better storage efficiency and data protection than Kafka DAS clusters that use a 3x
replication factor. This allows Kafka administrations to safely lower the Kafka replication factor and
increase storage capacity and efficiency with ease when using Isilon.

Appendix

Kafka Server Properties
Below is the running server property configuration file used for this paper, the only difference between
the Kafka DAS configuration and the Kafka Isilon configuration is what is highlighted below, everything
else is the same on both clusters.
broker.id=0
listeners=PLAINTEXT://k0:9092
num.network.threads=24
num.io.threads=8
socket.send.buffer.bytes=13107200
socket.receive.buffer.bytes=13107200
socket.request.max.bytes=104857600
# Just a single Isilon NFS mount needed with the Isilon config
log.dirs=/mnt/k0/kafka-logs
# Kafka DAS config has all direct attached disk drives (24) used, the remaining drive is for OS.
log.dirs=/data1/kafka/kafka-logs,/data2/kafka/kafka-logs,/data3/kafka/kafka-logs,/data4/kafka/kafka-
logs,/data5/kafka/kafka-logs,/data6/kafka/kafka-logs,/data7/kafka/kafka-logs,/data8/kafka/kafka-
logs,/data17/kafka/kafka-logs,/data18/kafka/kafka-logs,/data19/kafka/kafka-logs,/data21/kafka/kafka-logs
,/data21/kafka/kafka-logs,/data22/kafka/kafka-logs,/data23/kafka/kafka-logs,/data24/kafka/kafka-logs
num.partitions=8
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=k0:2181,k1:2181,k2:2181,k3:2181,k4:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0
delete.topic.enable=true

Zookeeper Properties
Below is the zookeeper properties file used for both Kafka DAS and Kafka Isilon clusters.
clientPort=2181
maxClientCnxns=0
server.0=k0:2888:3888
server.1=k1:2888:3888
server.2=k2:2888:3888
server.3=k3:2888:3888
server.4=k4:2888:3888
initLimit=5
syncLimit=2
Producer Properties
bootstrap.servers=k0:9092,k1:9092,k2:9092,k3:9092,k4:9092
compression.type=lz4
Consumer Properties
bootstrap.servers=k0:9092,k1:9092,k2:9092,k3:9092,k4:9092
group.id=test-consumer-group

NFS Client Configuration
The Kafka Isilon cluster uses NFS to mount the remote Isilon OneFS file system. Using NFS centralizes all
the Kafka data on Isilon, this provides Kafka administrators an easy way to immediately increase storage
capacity on the fly by simply adding more Isilon nodes. No need to add servers or modify any of the
Kafka configuration, as soon as new Isilon nodes are added to the Isilon cluster, the Kafka cluster will
immediately see an increase in storage capacity and performance.
Since NFS is being use as the protocol to network the Kafka servers to Isilon, it’s important to optimize the
NFS client settings to obtain the performance detailed in this paper. Below are the NFS mount options
used in /etc/fstab on each Kafka server for the Kafka Isilon cluster.
Note: The mount options are the same on each Kafka server, the only difference on each server is the
mount point itself. The below example is the configuration for Kafka server k0 only, which mounts /ifs/k0
from Isilon. Kafka server k1 mounts /ifs/k1 from Isilon and so on. Each Kafka server needs its own unique
NFS export from Isilon/OneFS. The Isilon configuration is describe in detail in the next section for
reference. A single export on Isilon could have been used as well, in this case, each Kafka server would
just mount a different sub directory.
Example Kafka Server K0 NFS Client Configuration:
/etc/fstab
isilon.example.com:/ifs/k0 /mnt/k0 nfs nolock, noacl, nocto, noatime, async, nodiratime,
nfsvers=3, tcp, rw, hard, intr, timeo=600, retrans=2, rsize=524288, wsize=524288 0 0
The isilon.example.com entry in /etc/fstab above corresponds to the OneFS SmartConnect Zone Name.
SmartConnect is what provides load balancing via DNS, so you must delegate this zone name to Isilon
on your DNS server to ensure a proper load balancing configuration for Kafka.
See the SmartConnect Whitepaper for further information.
A breakdown of what the NFS mount options above do are described below for reference. These
settings increase NFS performance.

async mode allows Isilon to reply to the NFS client as soon as it has processed the I/O request and sent
it to the local filesystem.
nfsvers=3 specifies NFS version 3 to be used.
noacl disables Access Control List (ACL) processing.
noatime option specifies that inode access times are not updated on the filesystem.
nocto option suppresses the retrieval of new attributes when creating a file.
nodiratime option specifies that the directory inode is not updated on the filesystem when it is
accessed.
nolock option prevents the exchange of file lock information between the NFS server and this NFS client.
retrans specifies the number of tries the NFS client will make to retransmit the packet.
rsize and wsize options specify the number of bytes per NFS read and write request.
rw option mounts the remote NFS file system in read/write mode.
tcp option specify NFS over TCP instead of UDP.
timeo option is the amount of time the NFS client waits on the NFS server before retransmitting a
packet.

Isilon Configuration
The following features were configured on the Isilon cluster. The Smart features shown below are
product differentiators that significantly enhance data storage performance and resiliency.
 Enable SmartPools settings across all Isilon nodes
 Enable SmartConnect to provide automatic client connection load balancing and failover
capabilities
 Enable SmartCache for write performance and Streaming Access for Data Access Optimization
 Use optimization for streaming data access pattern
 Use a 40 Gb/s external network ports for NFS connections and internal 40 Gb/s ports for data
interconnect network
 Increase network MTU to 9000 (Jumbo Frames) for both internal and external networks
The SmartCache and Streaming Access optimizations are easily enabled in the Isilon OneFS GUI through
a File Pool Policy tab. The Kafka Isilon cluster screen-shot is shown below for reference.

The Storage Pool is created on the SmartPools tab, this allows you to specify the Isilon nodes and
protection settings. Below is a screen-shot of the storage pool configured for the Kafka Isilon cluster.
The internal and external network configuration is set in the network configuration tab, here you specify
MTU size, IP info, and DNS server info.

The SmartConnect info is configured within the pool properties, in the Kafka Isilon cluster, the pool name
is pool0 and the SmartConnect info (IP and DNS info blacked-out) is shown in the screen-shot below:
Note: Isilon provides 2 x 40 GbE front-end and 2 x 40GbE back-end ports with each node, the Kafka
Isilon cluster was only configured with one 40 GbE front-end port on each node during performance
testing. With production deployments, use both front-end 40 GbE ports.
Lastly, the NFS exports are configured for each Kafka server, namely /ifs/k0 to /ifs/k11 as shown below:

The export settings for NFS v 3 are in the Global Settings tab:

OneFS TCP Tuning
The default TCP stack of OneFS needs some tuning for Kafka and 40GbE connectivity. The tuning needs
to be done within the CLI directly on Isilon. A tcptune.sh script is available at Github.
Simply run sh ./tcptune.sh Max to make the changes, an example script run is shown below:
Before changes:
isilon# sh ./tcptune.sh Max
Tuning TCP stack to Max
TCP sysctls
before...
kern.ipc.maxsockbuf=2097152
net.inet.tcp.sendbuf_max=2097152
net.inet.tcp.recvbuf_max=2097152
net.inet.tcp.sendbuf_inc=8192
net.inet.tcp.recvbuf_inc=16384
net.inet.tcp.sendspace=131072
net.inet.tcp.recvspace=131072
efs.bam.coalescer.insert_hwm=209715200
efs.bam.coalescer.insert_lwm=178257920
After Changes: Apply tuning...
Value set successfully
TCP sysctls after...
kern.ipc.maxsockbuf=104857600
net.inet.tcp.sendbuf_max=52428800

net.inet.tcp.recvbuf_max=52428800
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=32768
net.inet.tcp.sendspace=26214400
net.inet.tcp.recvspace=26214400
efs.bam.coalescer.insert_hwm=209715200
efs.bam.coalescer.insert_lwm=178257920
net.inet.tcp.mssdflt=8948
That’s basically it for the Isilon configuration. With Dell EMC Isilon Scale-out NAS, you can now deploy
your own Kafka cluster and centrally store all your data while supporting millions of Kafka write and read
operations a second.

Kafka End-to-End Latency Test
Latency information is provided in the results of some of the tests runs, however the Kafka results were
not end-to-end latency results.
Kafka provides a latency tool to test end-to-end latency between Kafka Producer and Kafka
Consumer. Below are the results of the end-to-end latency test for a single Producer and single
Consumer for both Kafka DAS and Kafka Isilon cluster. The 99.9th percentile latency is better with Isilon
F800. This tool does not provide throughput information. The test is for a 100 byte record size and 5000
records.
DAS bin/kafka-run-class.sh kafka.tools.EndToEndLatency k0:9092 DAS-latency 5000 all 100
0 204.818597
1000 2.255263
2000 1.697824
3000 1.760031
4000 1.704499
Avg latency: 2.7261 ms
Percentiles: 50th = 1, 99th = 36, 99.9th = 50
F800 bin/kafka-run-class.sh kafka.tools.EndToEndLatency k0:9092 F800-latency 5000 all 100
0 158.756064
1000 1.97554
2000 2.1549609999999997
3000 1.612731
4000 1.8153979999999998
Avg latency: 3.7892 ms
Percentiles: 50th = 2, 99th = 36, 99.9th = 41

Apache Kafka Performance with Dell EMC Isilon F800 All-Flash NAS

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Apache Kafka Performance with Dell EMC Isilon F800 All-Flash NAS

Similar to Apache Kafka Performance with Dell EMC Isilon F800 All-Flash NAS (20)

More from Boni Bruno

More from Boni Bruno (8)

Recently uploaded

Recently uploaded (20)

Apache Kafka Performance with Dell EMC Isilon F800 All-Flash NAS