Kosovo 2009

1. Kosova Sofware Freedom
Conference 2009
Clusters with GlusterFS
Marian Marinov - mm@yuhu.biz
System Architect - Siteground.com
Prishtina 29-30.Aug.2009

2. Agenda
Cluster Filesystems
Some facts
Gluster Design
➢ kernel
➢ gluster engine
➢ protocols
➢ translators
➢ storage
➢ performance
➢ others
➢ schedulers
Some benchmarks
2
Prishtina 29-30.Aug.2009 1/29

3. Cluster Filesystems
3
Prishtina 29-30.Aug.2009

4. Cluster Filesystems
4
Prishtina 29-30.Aug.2009 3/29

5. Cluster Filesystems
5
Prishtina 29-30.Aug.2009

6. GFarm Desgin
6
Prishtina 29-30.Aug.2009 6/29

7. Facts
GlusterFS project starts in August 2006
➢ It is not actual Filesystem
➢ Server only for any POSIX compliant
but mainly tested on Linux
➢ Client running on Linux, FreeBSD & MacOS X
as they require FUSE
➢ Very scallable
➢ Very easy to install and maintain
7
Prishtina 29-30.Aug.2009 4/29

8. GlusterFS Desgin
8
Prishtina 29-30.Aug.2009

9. Gluster Filesystem Design
In the kernel
➢ Requires FUSE
➢ FUSE as module
➢ GlusterFUSE
The engine
➢ Server & Client
➢ Transport Modules
➢ Translators
➢ Scheduler Modules
9
Prishtina 29-30.Aug.2009

10. GlusterFS Desgin
10
Prishtina 29-30.Aug.2009

11. GlusterFS Desgin
11
Prishtina 29-30.Aug.2009

12. GlusterFS Desgin
12
Prishtina 29-30.Aug.2009

13. GlusterFS Desgin
13
Prishtina 29-30.Aug.2009

14. GlusterFS Desgin
The picture explained:
ClientX:
volume serverX - defines a name for a remote server
subvolumes brick0 - defines in which of all exported volumes from
the remote server we are interested
some performance translators
volume unify - defines that we will use unify cluster translator
subvolumes serverX serverY - defines which already connected storage
volumes will be used
14
Prishtina 29-30.Aug.2009

15. Gluster Filesystem Design
Transport Modules:
For TCP/IP transport
transport-type tcp/server
For Infiniband SDP transport
transport-type ib-sdp/server
For Infiniband Verbs transport
transport-type ib-verbs/server
15
Prishtina 29-30.Aug.2009

16. Gluster Filesystem Design
The idea – GNU/Hurd
Translators
➢ Performance
➢ Clustering
➢ Features
➢ Storage
➢ Others
16
Prishtina 29-30.Aug.2009 14/29

17. Gluster Filesystem Design
Performance translators
➢ Read Ahead
➢ Write Behind
➢ Threaded I/O
➢ IO-Cache
➢ Stat Pre-fetch – still not ported to the new versions
➢ Booster
17
Prishtina 29-30.Aug.2009 15/29

18. Gluster Filesystem Design
Clustering translators
➢ Distributed Hash Table (DHT)
➢ Stripe
➢ Replicate (old AFR)
➢ Unify (new HA)
➢ NUFA
18
Prishtina 29-30.Aug.2009 16/29

19. Gluster Filesystem Design
Distributed Hash Table (DHT)
➢ lookup-unhashed
➢ min-free-disk
Replicate
➢ read-subvolume
➢ favorite-child
➢ data-self-heal, metadata-self-heal & entry-self-heal
➢ data-change-log', metadata-change-log & entry-change-log
➢ data-lock-server-count, metadata-lock-server-count
& entry-lock-server-count
Stripe & Unify
19
Prishtina 29-30.Aug.2009

20. Gluster Filesystem Design
Scheduling systems
➢ Adaptive Least Usage (ALU)
➢ Non-uniform filesystem architecture (NUFA)
➢ Random
➢ Rand-Robin
➢ Switch
20
Prishtina 29-30.Aug.2009 17/29

21. Gluster Filesystem Design
Adaptive Least Usage (ALU)
➢ disk-usage
➢ read-usage
➢ write-usage
➢ open-files-usage
➢ disk-speed-usage
21
Prishtina 29-30.Aug.2009 18/29

22. Gluster Filesystem Design
Non-uniform filesystem architecture (NUFA)
➢ local-volume-name
➢ limits.min-free-disk
Random
➢ limits.min-free-disk
Round-Robin
➢ limits.min-free-disk
➢ read-only-subvolumes
➢ refresh-interval
22
Prishtina 29-30.Aug.2009 19/29

23. Gluster Filesystem Design
Switch
➢ switch.case *jpg:brick1,brick2;*mp3:brick3;*:brick4,brick5
➢ switch.read-only-subvolumes brick7
23
Prishtina 29-30.Aug.2009 20/29

24. Gluster Filesystem Design
Other translators
➢ client
➢ server
➢ posix
➢ posix-locks
➢ bdb
➢ filter
➢ rot-13
➢ trace
24
Prishtina 29-30.Aug.2009 21/29

25. Gluster Filesystem Design
filter
➢ root-squashing
➢ read-only
➢ fixed-uid & fixed-gid
➢ translate-uid & translate-gid
➢ filter-uid & filter-gid
25
Prishtina 29-30.Aug.2009 21/29

26. Gluster Filesystem Design
In the feature
➢ Live addition/removal of nodes
➢ Automatic File Reordering
➢ Web GUI
➢ mod_glusterfs
26
Prishtina 29-30.Aug.2009 22/29

27. Gluster Design
27
Prishtina 29-30.Aug.2009 23/29

28. Benchmarks
28
Prishtina 29-30.Aug.2009 24/29

29. Benchmarks
29
Prishtina 29-30.Aug.2009

30. Benchmarks
Aggregated Read Throughput Benchmark
Multiple dd utility were executed simultaneously with different block sizes
to read from GlusterFS filesystem.
4KB 16KB 128KB 256KB 512KB 1024KB
Lustre 1,796 MB/s 5,782 MB/s 20,423 MB/s 21,582 MB/s 22,789 MB/s 23,731 MB/s
GlusterFS 11,415 MB/s 11,424 MB/s 11,427 MB/s 11,419 MB/s 11,411 MB/s 11,409 MB/s
Aggregated Write Throughput Benchmark
Multiple dd utility were executed simultaneously with different block sizes
to write to GlusterFS filesystem.
4KB 16KB 128KB 256KB 512KB 1024KB
Lustre 969 MB/s 1,613 MB/s 1,988 MB/s 1,989 MB/s 1,984 MB/s 1,983 MB/s
GlusterFS 1,886 MB/s 2,191 MB/s 2,237 MB/s 2,231 MB/s 2,236 MB/s 2,223 MB/s
Note: Higher means faster.
30
Prishtina 29-30.Aug.2009 26/29

31. Benchmarks
Apache Web Server Benchmark
Apache served 12039 files (595 MB) over HTTP protocol.
wget client fetched the files recursively.
Time
Lustre Failed after downloading 33 MB out of 585 MB in 11 mins.
GlusterFS 3 mins 11 secs
Archive Creation
'tar utility created an archive of 12039 files (595 MB) served through GlusterFS.
Time
Lustre 41 secs
GlusterFS 25 secs
Archive Extraction
Time
Lustre FAILED No space left on device.
GlusterFS 43 secs
Note: Lower means faster.
31
Prishtina 29-30.Aug.2009 27/29

32. Benchmarks
32
Prishtina 29-30.Aug.2009

33. Benchmarks
Sequencial Write : 1KB x 1,000,000 times = 1GB
# time dd if=/dev/zero of=/mnt/unify/file bs=1024 count=1000000
Test Case Local SATA Local NFS Single Unified
500G RAID0 GlusterFS GlusterFS
1 Worker - 11.836s 11.371s 23.162s 2m19.597s 3m39.279s
1st test (82.5MB/s) (85.9MB/s) (42.2MB/s) (7.0MB/s) (4.4MB/s)
1 Worker - 10.537s 10.777s 24.181s 2m24.623s 3m40.334s
2nd test (92.7MB/s) (90.6MB/s) (40.4MB/s) (6.7MB/s) (4.4MB/s)
33
Prishtina 29-30.Aug.2009

34. Benchmarks
Sequencial Write : 64KB x 15,625 times = 1GB
# time dd if=/dev/zero of=/mnt/unify/file bs=65536 count=15625
Test Case Local SATA Local NFS Single Unified
500G RAID0 GlusterFS GlusterFS
1 Worker - 6.390s 7.939s 22.766s 24.637s 22.436s
1st test (152.8MB/s) (123.0MB/s) (42.9MB/s) (39.6MB/s) (43.5MB/s)
1 Worker - 6.588s 7.542s 21.901s 22.001s 23.378s
2nd test (148.2MB/s) (129.5MB/s) (44.6MB/s) (44.4MB/s) (41.8MB/s)
34
Prishtina 29-30.Aug.2009

35. Sources of Information
Project's site:
http://www.gluster.com
Official GlusterFS documentation wiki:
http://www.gluster.org/docs/index.php/GlusterFS
On IRC:
irc.freenode.net #gluster
The mailing list:
gluster-devel@nongnu.org
35
Prishtina 29-30.Aug.2009 28/29

36. Clusters with GlusterFS
? ? ? ? ? ? ?
? ? ?? ? ? ?
? ? Questions ? ?
? ? ? ? ? ? ?
? ? ? ? ? ?
? ? ? ? ? ? ? 36
Prishtina 29-30.Aug.2009