Gluster performance was analyzed for different workload classes and volume types. Replica volumes performed better than erasure coded volumes for low thread count sequential workloads, especially writes, while erasure coded volumes excelled for writes with higher thread counts. Kernal NFS generally outperformed Gluster for single-threaded sequential workloads but Gluster performed comparably or better for workloads with more concurrent threads. Random read performance was similar between volume types but replica volumes suffered on random writes due to the use of RAID-6. Small file performance is still being investigated. Erasure coding shows potential for lower cost video storage use cases.

1. State of Gluster
Performance
Manoj Pillai - mpillai@redhat.com
Red Hat Perfomance and Scale Engineering

2. Agenda
● Motivation
● Gluster performance by workload class
● EC for video storage use-cases
● Work in Progress

3. Motivation
● Understand gluster performance characteristics, for
different workload classes and workload parameters
● Identify areas for improvement

4. Configuration
● Systems:
○ 12 drives per server
○ Replica-2 results with single 12-disk RAID-6 device
on each server
○ EC results with brick on each drive
○ Upto 6 servers, 4 clients in these tests
○ 1 10GbE link each
● Software versions
○ Glusterfs-3.8.2-1.el7.x86_64, RHEL-7
● Standard tuning
○ lookup-optimize, client-io-threads: on
○ event-threads=4

5. Sequential I/O: Replica
Volume
● With sufficient number of requests concurrently in flight,
close to network line speed on both writes and reads.

6. Sequential I/O: Disperse
Volume
● Similar to dist-rep, but larger gap in throughput between
low thread count and high thread count.

7. Seq I/O: Comparison of
Volume Types
● Replica-2 performs better for low thread count,
especially on writes.
● Disperse excels for writes with higher no. of threads.

8. Seq Write: Comparison with
kNFS
● kNFS better on single-thread tests, gluster does well with
higher number of threads.

9. Seq Read: Comparison with
kNFS
● kNFS dominates on single-thread test, hits line speed

10. Random Read: Comparison
with kNFS
● “1-cached” is single thread test where server-side cache
has not been dropped.
○ Good test of latency in the I/O path

11. Random Write: Comparison
with kNFS
● All of these tests bottleneck on disk
○ RAID-6 not a good choice for random writes.

12. Random Read: Comparison of
Volume Types

13. Random Write: Comparison
of Volume Types
● Replica-2 suffers possibly because of RAID-6 poor
performance on random writes.

14. Small File Read: Comparison
of Volume Types
● Disperse at a clear disadvantage in these tests

15. Small File Create:
Comparison of Volume Types
● Small file performance still under investigation

16. EC for Video Storage
Use-Cases
● Lot of interest seen in this area
● Significantly lower cost for EC, good performance

17. Work In Progress
● Rebalance Performance
○ Crawl, migration and throttle settings
● Small-file Performance
○ Performance degradation with increasing number of
files
● Tiering performance for small files
○ Small file performance with EC cold tier

18. Theme
Thanks!!