Trying and evaluating the new features of GlusterFS 3.5

Copyright （C） 2014, NTTPC Communications, Inc. All Rights Reserved. 2　
Agenda
!  About me
!  The new features
!  Additional news
!  Conclusion

About
Me

About me
!  Work at as a ...
!  Programmer
!  (Most recently)
! Also Interested in
!  Software Engineer
! Cloud Computing
! Big Data/Data Science
! Something new technologies
! Supporting GlusterFS/Red Hat Storage Introduction
with Red Hat K.K.

About me
! GlusterFS since 2007 (v1.3.7)
!  for my internet crawler at first.
!  Love Gluster because of the ...
!  Potential
!  Performance
!  Code
!  Community
!  Introduced or introducing it into ...
!  Printer and scanner solution (field trial)
!  Email services
!  File storage services (WebDAV, NFS)
!  Backup services
!  Shared storage platform
!  Medical service
!  A board member of the Gluster
Community

My system
Seg.1: 192.168.79.0/24, GigE
Seg.2: 10.0.0.0/8, 100BaseT(USB Ethernet)
eins
zwei
drei
vier
fuenf
sechs
sieben
.1
.2
.3
.4
.5
.6
.7
.79.0.1
.79.0.2
.79.0.3
.79.0.4
.79.0.5
.79.0.6
•  Seven nodes, connected to two separated physical network
segments.
•  Seg.1 is for GlusterFS and Seg.2 is for other purposes (e.g. SSH)
•  Each node is setup with:
•  CentOS 6.5 x86_64
•  GlusterFS 3.5.0 (from source tarball)
.79.0.7
storage
pool
(mainly)
client

My system
•  Intel NUC DN2820FYKH
•  Celeron2.4GHz dual-core, 1MB cache
•  8GB RAM
•  1TB Solid-state hard drive (w/ 8GB SLC SSD)
•  7.5W TDP
•  Why?
•  Separate several
loads (mainly of disk
accesses and
network traffics)
•  Enough cheap to
build (38k JPY/node)
•  Save money on
electricity (2 JPY/d/
node)
•  Suppress my room's
temperature
increasing

My system
% sudo yum install -y openssh-clients make rpm-build bison flex automake libtool ncurses-
devel readline-devel openssl-devel libxml2-devel libibverbs-devel libacl-devel libattr-devel
python-devel python-setuptools lvm2-devel systemtap-sdt-devel libaio-devel xfsprogs glib2-
devel
% tar xzf glusterfs-3.5.0.tar.gz && cd glusterfs-3.5.0
% ./configure --prefix=/usr/local/glusterfs-3.5.0 --enable-bd-xlator --enable-fusermount --
enable-systemtap --enable-debug --enable-crypt-xlator --enable-qemu-block --enable-glupy
% make && sudo make install
# ln -sfn /usr/local/glusterfs-3.5.0 /usr/local/glusterfs
# cp -p /etc/init.d/glusterd /etc/init.d/glusterd-3.5.0
# cat <<EOF >> ~/.zshrc
export PATH=$PATH:/usr/local/glusterfs/sbin
export MANPATH=$MANPATH:/usr/local/glusterfs/share/man
EOF
# source ~/.zshrc
# echo "/usr/local/glusterfs/lib" > /etc/ld.so.conf.d/glusterfs.conf
# ldconfig
# sed -i 's/SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
# chkconfig iptables off
# /etc/init.d/iptables stop
GlusterFS 3.5.0 was installed on each node
in following way:

12 new features

Overview
Features Open
Stack
Opera-
tion
Mana-
gement
Scala-
bility
Perfor-
mance
Stab-
ility
Sec-
urity
Dev
AFR_CLI_enhancements ✔️
Exposing Volume
Capabilities
✔️
File Snapshot ✔️
GFID Access ✔️
On-Wire Compression +
Decompression
✔️
Prevent NFS restart on
Volume change (Part 1)
✔️
Quota Scalability ✔️ ✔️
readdir_ahead ✔️
zerofill ✔️ ✔
Brick Failure Detection ✔️
Disk encryption ✔️
Geo-Replication
Enhancement
✔ ✔️

OpenStack
Integration
Enhancements

File Snapshot
# setfattr -n
trusted.glusterfs.block-format
-v qcow2:<file_size(in KB/MB/GB)>
<file_name>
features/
qemu-block
xlator
# setfattr -n
trusted.glusterfs.block-snapshot-create
-v <snapshot_name1> <file_name>
# setfattr -n
trusted.glusterfs.block-snapshot-create
# setfattr -n
trusted.glusterfs.block-snapshot-goto
# setfattr -n
trusted.glusterfs.block-snapshot-delete
a file
<file_name>
under
a mount
point
of a volume
fuse
hook
to
glusterfs
client
process
Restore from a snapshot
Take a snapshot
Take a snapshot

File Snapshot
features/
qemu-block
xlator
a file
<file_name>
under
a mount
point
of a volume
as a block
storage
for
Cinder
fuse
hook
to
glusterfs
client
process
Restore from a snapshot
Take a snapshot
Take a snapshot
OpenStack
Cinder
BD
xlator
block-format
block-snapshot-create
block-snapshot-create
block-snapshot-goto
block-snapshot-delete

zerofill
glusterfsdglusterfsd
AFR
libgfapi
User App
(e.g. Cinder)
0000
0000
0000
0000
0000
0000
posix_do_zerofill function
ZEROFILL fop
(glfs_zerofill function)
SCSI WRITESAME
command
BLKZEROOUT ioctl on Linux

zerofill
Server offloaded zerofill vs repeated zeroing
[root@llmvm02 remote]# time ./offloaded aakash-test log 20
real 3m34.155s
user 0m0.018s
sys 0m0.040s
[root@llmvm02 remote]# time ./manually aakash-test log 20
real 4m23.043s
user 0m2.197s
sys 0m14.457s
[root@llmvm02 remote]# time ./offloaded aakash-test log 25;
real 4m28.363s
user 0m0.021s
sys 0m0.025s
[root@llmvm02 remote]# time ./manually aakash-test log 25
real 5m34.278s
user 0m2.957s
sys 0m18.808s
http://www.gluster.org/community/documentation/index.php/Features/zerofill
1.23 times faster!
1.25 times faster!

Operation
Enhancements

AFR_CLI_enhancements
Before 3.5.0
# gluster volume heal vol1
Heal operation on volume vol1 has been successful
# gluster volume heal vol1 info
...
# gluster volume heal vol1 info healed
...
# gluster volume heal vol1 info heal-failed
...
# gluster volume heal vol1 info split-brain
...
Too many operations to know all
the situations...
What I want to know is not the
file names...
How long the healing takes?
I don't know when the split-brain
detected but...

AFR_CLI_enhancements
After 3.5.0
# gluster volume heal vol1 statistics
Gathering crawl statistics on volume vol1 has been successful
------------------------------------------------
Crawl statistics for brick no 0
Hostname of brick eins
Starting time of crawl: Mon May 19 10:13:02 2014
Ending time of crawl: Mon May 19 10:13:02 2014
Type of crawl: INDEX
No. of entries healed: 0
No. of entries in split-brain: 0
No. of heal failed entries: 0
...
Wow! I can get the statistic and historical
information at a glance!

Management
Enhancements

# gluster volume info
Volume Name: bd0
Type: Distribute
Volume ID: 019d0f4b-d11a-480e-9be8-0c79902f0746
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: sieben:/tmp/bd0-meta
Exposing Volume Capabilities
I confuse which volume type the volume supports.
So I should manage it with other tools like Excel...
Before 3.5.0

Volume Name: bd0
Type: Distribute
Status: Started
Xlator 1: BD
Capability 1: thin
Capability 2: offload_copy
Capability 3: offload_snapshot
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1 VG: bd0-vg
Exposing Volume Capabilities
Probe the type of
volume
Provide list of
capabilities of a
xlator/volume.
Yeah! I can understand the volume type
and the detail!
After 3.5.0

Review: How to use BD xlator
# dd if=/dev/zero of=/tmp/bd-loop6 bs=1M count=2048
# losetup /dev/loop6 /tmp/bd-loop6
# pvcreate /dev/loop6
# vgcreate bd0-vg /dev/loop6
Volume group "bd0-vg" successfully created
# lvcreate --thin bd0-vg -L 1000M
Logical volume "lvol0" created
Logical volume "lvol1" created
This VG becomes a
volume of GlusterFS
If you want to get the BDs
thin-provisioned ones, hit
the lvcreate command.
(And the names are fixed.)
Here created a VG with
a single 2GB of PV

# lvdisplay bd0-vg
--- Logical volume ---
LV Name lvol1
VG Name bd0-vg
LV UUID PSAFkr-Vyr8-fkGU-kDnA-rWUF-fFFT-111Snr
LV Write Access read/write
LV Creation host, time sieben, 2014-05-18 14:38:21 +0900
LV Pool transaction ID 0
LV Pool metadata lvol1_tmeta
LV Pool data lvol1_tdata
LV Pool chunk size 64.00 KiB
LV Zero new blocks yes
LV Status available
# open 0
LV Size 1000.00 MiB
Allocated pool data 0.00%
Allocated metadata 0.88%
Current LE 250
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:5
A logical volume pool for
thin-provisioning.
No need when using no
thin-provisioning.

# mkdir /tmp/bd0-meta
# gluster volume create bd0 sieben:/tmp/bd0-meta?bd0-vg force
volume create: bd0: success: please start the volume to access data
# gluster volume start bd0
volume start: bd0: success
# gluster volume info bd0
Volume Name: bd0
Type: Distribute
Status: Started
Xlator 1: BD
Capability 1: thin
Capability 2: offload_copy
Capability 3: offload_snapshot
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1 VG: bd0-vg
# mkdir /mnt/glusterfs/bd0
# mount-t glusterfs sieben:/bd0 /mnt/glusterfs/bd0
Meta data store
for BD xlator
"?" (question
mark) is the
separator

# touch /mnt/glusterfs/bd0/lv0
# setfattr -n "user.glusterfs.bd" -v "thin:1024MB" /mnt/glusterfs/bd0/lv0
# lvdisplay bd0-vg
LV Name lvol1
VG Name bd0-vg
LV UUID PSAFkr-Vyr8-fkGU-kDnA-rWUF-fFFT-111Snr
LV Creation host, time sieben.infinibridge.net, 2014-05-18 14:38:21 +0900
LV Pool transaction ID 1
LV Pool metadata lvol1_tmeta
LV Pool data lvol1_tdata
LV Pool chunk size 64.00 KiB
LV Zero new blocks yes
LV Status available
# open 0
LV Size 1000.00 MiB
Allocated pool data 0.00%
Allocated metadata 0.98%
Current LE 250
Segments 1
Allocation inherit
Block device 253:5
Create a file that
is backed by an
LV
Or simply -v "lv" when
no need for thin-
provisioning

LV Path /dev/bd0-vg/a9790eba-ffbf-4d9c-a674-e02c61ece935
LV Name a9790eba-ffbf-4d9c-a674-e02c61ece935
VG Name bd0-vg
LV UUID Z4HtWM-W0jk-YiK5-66ED-zOMw-YhFp-nrnRUU
LV Creation host, time sieben.infinibridge.net, 2014-05-18 14:47:31 +0900
LV Pool name lvol1
LV Status available
# open 0
LV Size 1.00 GiB
Mapped size 0.00%
Current LE 256
Segments 1
Allocation inherit
Block device 253:9

# for i in `seq 1 9`; do touch /mnt/glusterfs/bd0/lv$i; setfattr -n "user.glusterfs.bd" -v
"thin:1024MB" /mnt/glusterfs/bd0/lv$i; done
# lvdisplay -C bd0-vg
LV VG Attr LSize Pool Origin Data% Move
Log Cpy%Sync Convert
39b82644-f8ef-435d-b14e-d199a7e264fa bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
6002ddb2-28f1-463c-8666-f683fe2441ed bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
69993340-d691-4502-a9d5-375b8be0fb9e bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
82af50a2-0124-41d8-a887-d8c30427a663 bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
996969dd-3e32-491b-95d1-f279e6808d5b bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
a19ac2af-94df-4d01-b7c3-bbfcbfe5d09e bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
a9790eba-ffbf-4d9c-a674-e02c61ece935 bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
d6fd964a-67f8-4d48-96d1-343bed4ee792 bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
ea58b011-3a41-4bf0-9fe6-3862e24b86f6 bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
f7df48e5-09b1-4314-b729-1f38e5ceec2e bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
lvol1 bd0-vg twi-a-tz-- 1000.00m 0.00
Here we create other
nine LVs in the same
way.

# mkdir /mnt/bd0-lv/{39b82644-f8ef-435d-b14e-d199a7e264fa,6002ddb2-28f1-463c-8666-
f683fe2441ed,69993340-d691-4502-a9d5-375b8be0fb9e,82af50a2-0124-41d8-a887-
d8c30427a663,996969dd-3e32-491b-95d1-f279e6808d5b,a19ac2af-94df-4d01-b7c3-
bbfcbfe5d09e,a9790eba-ffbf-4d9c-a674-
e02c61ece935,d6fd964a-67f8-4d48-96d1-343bed4ee792,ea58b011-3a41-4bf0-9fe6-3862e24b86f6,f7df4
8e5-09b1-4314-b729-1f38e5ceec2e}
# ls /mnt/bd0-lv
39b82644-f8ef-435d-b14e-d199a7e264fa a19ac2af-94df-4d01-b7c3-bbfcbfe5d09e
6002ddb2-28f1-463c-8666-f683fe2441ed a9790eba-ffbf-4d9c-a674-e02c61ece935
69993340-d691-4502-a9d5-375b8be0fb9e d6fd964a-67f8-4d48-96d1-343bed4ee792
82af50a2-0124-41d8-a887-d8c30427a663 ea58b011-3a41-4bf0-9fe6-3862e24b86f6
996969dd-3e32-491b-95d1-f279e6808d5b f7df48e5-09b1-4314-b729-1f38e5ceec2e
# for x in 39b82644-f8ef-435d-b14e-d199a7e264fa 6002ddb2-28f1-463c-8666-f683fe2441ed
69993340-d691-4502-a9d5-375b8be0fb9e 82af50a2-0124-41d8-a887-d8c30427a663
996969dd-3e32-491b-95d1-f279e6808d5b a19ac2af-94df-4d01-b7c3-bbfcbfe5d09e a9790eba-
ffbf-4d9c-a674-e02c61ece935 d6fd964a-67f8-4d48-96d1-343bed4ee792
ea58b011-3a41-4bf0-9fe6-3862e24b86f6 f7df48e5-09b1-4314-b729-1f38e5ceec2e; do mkfs.xfs -i
size=512 /dev/bd0-vg/$x && mount -t xfs /dev/bd0-vg/$x /mnt/bd0-lv/$x; done
Creating mount
point for each LV.
Formatting each LV
in XFS and mount it.

# df -h | grep bd0-lv
/dev/dm-13 1014M 33M 982M 4% /mnt/bd0-lv/39b82644-f8ef-435d-b14e-d199a7e264fa
/dev/dm-16 1014M 33M 982M 4% /mnt/bd0-lv/6002ddb2-28f1-463c-8666-f683fe2441ed
/dev/dm-18 1014M 33M 982M 4% /mnt/bd0-lv/69993340-d691-4502-a9d5-375b8be0fb9e
/dev/dm-11 1014M 33M 982M 4% /mnt/bd0-lv/82af50a2-0124-41d8-a887-d8c30427a663
/dev/dm-12 1014M 33M 982M 4% /mnt/bd0-lv/996969dd-3e32-491b-95d1-f279e6808d5b
/dev/dm-17 1014M 33M 982M 4% /mnt/bd0-lv/a19ac2af-94df-4d01-b7c3-bbfcbfe5d09e
/dev/dm-9 1014M 33M 982M 4% /mnt/bd0-lv/a9790eba-ffbf-4d9c-a674-e02c61ece935
/dev/dm-14 1014M 33M 982M 4% /mnt/bd0-lv/d6fd964a-67f8-4d48-96d1-343bed4ee792
/dev/dm-15 1014M 33M 982M 4% /mnt/bd0-lv/ea58b011-3a41-4bf0-9fe6-3862e24b86f6
/dev/dm-10 1014M 33M 982M 4% /mnt/bd0-lv/f7df48e5-09b1-4314-b729-1f38e5ceec2e
# mount | grep bd0-lv
/dev/mapper/bd0--vg-39b82644--f8ef--435d--b14e--d199a7e264fa on /mnt/bd0-lv/39b82644-f8ef-435d-b14e-d199a7e264fa type xfs (rw)
/dev/mapper/bd0--vg-6002ddb2--28f1--463c--8666--f683fe2441ed on /mnt/bd0-lv/6002ddb2-28f1-463c-8666-f683fe2441ed type xfs (rw)
/dev/mapper/bd0--vg-69993340--d691--4502--a9d5--375b8be0fb9e on /mnt/bd0-lv/69993340-d691-4502-a9d5-375b8be0fb9e type xfs (rw)
/dev/mapper/bd0--vg-82af50a2--0124--41d8--a887--d8c30427a663 on /mnt/bd0-lv/82af50a2-0124-41d8-a887-d8c30427a663 type xfs (rw)
/dev/mapper/bd0--vg-996969dd--3e32--491b--95d1--f279e6808d5b on /mnt/bd0-lv/996969dd-3e32-491b-95d1-f279e6808d5b type xfs (rw)
/dev/mapper/bd0--vg-a19ac2af--94df--4d01--b7c3--bbfcbfe5d09e on /mnt/bd0-lv/a19ac2af-94df-4d01-b7c3-bbfcbfe5d09e type xfs (rw)
/dev/mapper/bd0--vg-a9790eba--ffbf--4d9c--a674--e02c61ece935 on /mnt/bd0-lv/a9790eba-ffbf-4d9c-a674-e02c61ece935 type xfs (rw)
/dev/mapper/bd0--vg-d6fd964a--67f8--4d48--96d1--343bed4ee792 on /mnt/bd0-lv/d6fd964a-67f8-4d48-96d1-343bed4ee792 type xfs (rw)
/dev/mapper/bd0--vg-ea58b011--3a41--4bf0--9fe6--3862e24b86f6 on /mnt/bd0-lv/ea58b011-3a41-4bf0-9fe6-3862e24b86f6 type xfs (rw)
/dev/mapper/bd0--vg-f7df48e5--09b1--4314--b729--1f38e5ceec2e on /mnt/bd0-lv/f7df48e5-09b1-4314-b729-1f38e5ceec2e type xfs (rw)
'Cause of thin-provisioning, in
total 10GB of block devices are
created on the 2GB of VG!

[sechs]# mount -t glusterfs localhost:/bd0 /mnt/glusterfs/bd0
[sechs]# mount -t xfs -o loop /mnt/glusterfs/bd0/lv0
[sechs]# df -h | grep bd0-lv
1014M 33M 982M 4% /mnt/bd0-lv/lv1
The block devices are shared
with GlusterFS as files.
raw block device
physical volume
volume group
LV LV LV
BD volume=
file file file
Convert them with lvm2
development library
=
Shared with
GlusterFS
Snapshot and
clone are
capable as LV

Brick Failure Detection
Before 3.5.0
1. One of the
backend storage
failed!
2. R/W ops from
a client
AFR
3. glusterfsd returned
"Input/output error" or
"Read-only filesystem"
directly.

After 3.5.0
1. One of the
backend storage
failed!
3. R/W ops from
a client
AFR
4. The client gets no
error and completes
the operation.
2. glusterfsd outputs logs
and shutdowns itself.

# brick="/mnt/lv4/vol4"; gluster volume create vol4 eins:$brick zwei:$brick drei:$brick
vier:$brick fuenf:$brick sechs:$brick
# gluster volume start vol4
# gluster volume set vol4 storage.health-check-interval 10
# gluster volume info vol4
Volume Name: vol4
Type: Distribute
Volume ID: 706122a9-44fc-4d1d-8c3b-97482d98b95c
Status: Started
Number of Bricks: 6
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv4/vol4
Brick2: zwei:/mnt/lv4/vol4
Brick3: drei:/mnt/lv4/vol4
Brick4: vier:/mnt/lv4/vol4
Brick5: fuenf:/mnt/lv4/vol4
Brick6: sechs:/mnt/lv4/vol4
Options Reconfigured:
storage.health-check-interval: 10
Setup for test

[sechs]# dmsetup table
vg0-swift: 0 209715200 linear 8:7 838862848
vg0-cinder: 0 209715200 linear 8:7 419432448
vg0-lv4: 0 209715200 linear 8:7 1468008448
vg0-lv3: 0 209715200 linear 8:7 1258293248
vg0-lv2: 0 209715200 linear 8:7 1048578048
vg0-lv1: 0 209715200 linear 8:7 209717248
vg0-lv0: 0 209715200 linear 8:7 2048
vg0-glance: 0 209715200 linear 8:7 629147648
Setup for test (contd.)

[sechs]# echo 0 209715200 error > dmsetup-error-target
[sechs]# dmsetup load vg0-lv4 dmsetup-error-target
[sechs]# dmsetup resume vg0-lv4
[sechs]# dmsetup table
vg0-swift: 0 209715200 linear 8:7 838862848
vg0-cinder: 0 209715200 linear 8:7 419432448
vg0-lv4: 0 209715200 error
vg0-lv3: 0 209715200 linear 8:7 1258293248
vg0-lv2: 0 209715200 linear 8:7 1048578048
vg0-lv1: 0 209715200 linear 8:7 209717248
vg0-lv0: 0 209715200 linear 8:7 2048
vg0-glance: 0 209715200 linear 8:7 629147648
Brick failure test

[2014-05-18 18:49:53.720594] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file
changed
[2014-05-18 18:50:04.238239] W [posix-helpers.c:1294:posix_health_check_thread_proc] 0-vol4-
posix: stat() on /mnt/lv4/vol4 returned: Input/output error
[2014-05-18 18:50:04.238328] M [posix-helpers.c:1314:posix_health_check_thread_proc] 0-vol4-
posix: health-check failed, going down
Message from syslogd@sechs at May 19 03:50:04 ...
glusterfsd: [2014-05-18 18:50:04.238328] M [posix-helpers.c:
1314:posix_health_check_thread_proc] 0-vol4-posix: health-check failed, going down
[2014-05-18 18:50:34.238551] M [posix-helpers.c:1319:posix_health_check_thread_proc] 0-vol4-
posix: still alive! -> SIGTERM
1319:posix_health_check_thread_proc] 0-vol4-posix: still alive! -> SIGTERM
[2014-05-18 18:50:34.238910] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib64/libc.so.
6(clone+0x6d) [0x7f1144ebab7d] (-->/lib64/libpthread.so.0(+0x79d1) [0x7f114554d9d1] (-->/
usr/local/glusterfs-3.5.0/sbin/glusterfsd(glusterfs_sigwaiter+0xf0) [0x4085af]))) 0-:
received signum (15), shutting down
var/log/glusterfs/bricks/mnt-lv4-vol4.log on the failed node

May 19 03:49:55 sechs kernel: XFS (dm-7): metadata I/O error: block 0x0
("xfs_buf_iodone_callbacks") error 5 buf count 4096
May 19 03:49:57 sechs kernel: XFS (dm-7): metadata I/O error: block 0x6400108
("xlog_iodone") error 5 buf count 4096
May 19 03:49:57 sechs kernel: XFS (dm-7): xfs_do_force_shutdown(0x2) called from line 1062
of file fs/xfs/xfs_log.c. Return address = 0xffffffffa04dd131
May 19 03:49:57 sechs kernel: XFS (dm-7): Log I/O Error Detected. Shutting down filesystem
May 19 03:49:57 sechs kernel: XFS (dm-7): Please umount the filesystem and rectify the
problem(s)
May 19 03:50:04 sechs glusterfsd: [2014-05-18 18:50:04.238328] M [posix-helpers.c:
May 19 03:50:27 sechs kernel: XFS (dm-7): xfs_log_force: error 5 returned.
May 19 03:50:34 sechs glusterfsd: [2014-05-18 18:50:34.238551] M [posix-helpers.c:
May 19 03:50:57 sechs kernel: XFS (dm-7): xfs_log_force: error 5 returned.
syslog on the failed node

# gluster volume status vol4
Status of volume: vol4
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick eins:/mnt/lv4/vol4 49160 Y 2925
Brick zwei:/mnt/lv4/vol4 49159 Y 440
Brick drei:/mnt/lv4/vol4 49152 Y 32500
Brick vier:/mnt/lv4/vol4 49152 Y 32657
Brick fuenf:/mnt/lv4/vol4 49152 Y 24517
Brick sechs:/mnt/lv4/vol4 N/A N N/A
NFS Server on localhost 2049 Y 29535
NFS Server on zwei N/A N N/A
NFS Server on vier N/A N N/A
NFS Server on drei N/A N N/A
NFS Server on eins N/A N N/A
NFS Server on fuenf N/A N N/A
NFS Server on sechs N/A N N/A
Task Status of Volume vol4
------------------------------------------------------------------------------
There are no active volume tasks
gluster volume status

# ps -ef | grep glusterfsd | grep -v grep | wc -l
0
processes on the failed node

[sechs]# service glusterd restart
restart glusterd (and glusterfsd) on the failed node
[2014-05-18 18:58:17.197872] I [glusterfsd.c:1959:main] 0-/usr/local/glusterfs-3.5.0/sbin/
glusterfsd: Started running /usr/local/glusterfs-3.5.0/sbin/glusterfsd version 3.5git (/usr/
local/glusterfs-3.5.0/sbin/glusterfsd -s sechs --volfile-id vol4.sechs.mnt-lv4-vol4 -p /var/
lib/glusterd/vols/vol4/run/sechs-mnt-lv4-vol4.pid -S /var/run/
23afc72b5ceddccd28b405b1cdf5b4df.socket --brick-name /mnt/lv4/vol4 -l /usr/local/
glusterfs-3.5.0/var/log/glusterfs/bricks/mnt-lv4-vol4.log --xlator-option *-posix.glusterd-
uuid=0765d288-a59b-4ccf-90ae-c3332c83dbf4 --brick-port 49152 --xlator-option vol4-
server.listen-port=49152)
[2014-05-18 18:58:17.205310] I [socket.c:3561:socket_init] 0-socket.glusterfsd: SSL support
is NOT enabled
[2014-05-18 18:58:17.205486] I [socket.c:3576:socket_init] 0-socket.glusterfsd: using system
polling thread
[2014-05-18 18:58:17.205880] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT
enabled
[2014-05-18 18:58:17.205949] I [socket.c:3576:socket_init] 0-glusterfs: using system polling
thread
[2014-05-18 18:58:18.834910] I [graph.c:254:gf_add_cmdline_options] 0-vol4-server: adding
option 'listen-port' for volume 'vol4-server' with value '49152'

[2014-05-18 18:58:18.834976] I [graph.c:254:gf_add_cmdline_options] 0-vol4-posix: adding
option 'glusterd-uuid' for volume 'vol4-posix' with value '0765d288-a59b-4ccf-90ae-
c3332c83dbf4'
[2014-05-18 18:58:18.837332] I [rpcsvc.c:2064:rpcsvc_set_outstanding_rpc_limit] 0-rpc-
service: Configured rpc.outstanding-rpc-limit with value 64
[2014-05-18 18:58:18.837510] W [options.c:848:xl_opt_validate] 0-vol4-server: option
'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with
correction
[2014-05-18 18:58:18.837572] I [socket.c:3561:socket_init] 0-tcp.vol4-server: SSL support is
NOT enabled
[2014-05-18 18:58:18.837601] I [socket.c:3576:socket_init] 0-tcp.vol4-server: using system
polling thread
[2014-05-18 18:58:18.838445] E [common-utils.c:93:mkdir_p] 0-: Failed due to reason Input/
output error
[2014-05-18 18:58:18.838505] I [mem-pool.c:539:mem_pool_destroy] 0-vol4-changelog: size=108
max=0 total=0
[2014-05-18 18:58:18.838533] E [xlator.c:403:xlator_init] 0-vol4-changelog: Initialization
of volume 'vol4-changelog' failed, review your volfile again
[2014-05-18 18:58:18.838561] E [graph.c:307:glusterfs_graph_init] 0-vol4-changelog:
initializing translator failed
[2014-05-18 18:58:18.838610] E [graph.c:502:glusterfs_graph_activate] 0-graph: init failed
restart glusterd (and glusterfsd) on the failed node (contd.)

[2014-05-18 18:58:18.839480] W [glusterfsd.c:1095:cleanup_and_exit] (-->/usr/local/
glusterfs-3.5.0/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x1b5) [0x7f2981c837d8] (-->/usr/
local/glusterfs-3.5.0/sbin/glusterfsd(mgmt_getspec_cbk+0x36a) [0x40cf77] (-->/usr/local/
glusterfs-3.5.0/sbin/glusterfsd(glusterfs_process_volfp+0x18a) [0x408bf2]))) 0-: received
signum (0), shutting down
restart glusterd (and glusterfsd) on the failed node (contd.)

Scalability
Enhancement

Quota Scalability
Before 3.5.0
Directory Quota limitation
= a few hundreds per volume

Quota Scalability
After 3.5.0
Directory Quota limitation
= 65536 per volume

Performance
Enhancements

On-Wire Compression + Decompression
FUSE client Storage pool
Write ops
Read ops
3. Transport
2. Compression
1. open
and
write
4. Decompression
and
write to disk
1. open
and
read
4. Decompression 3. Transport
2. read and
Compression

# gluster volume create vol-comp eins:/mnt/lv3/vol-comp
# gluster volume set vol-comp network.compression on
# gluster volume set vol-comp network.compression.compression-level 8
# gluster volume set vol-comp network.compression.min-size 50
# gluster volume set vol-comp performance.write-behind off
# gluster volume set vol-comp performance.strict-write-ordering on
# gluster volume set vol-comp performance.open-behind off
# gluster volume info vol-comp
Volume Name: vol-comp
Type: Distribute
Volume ID: 92b47734-2552-4168-b3c3-151093562e4f
Status: Created
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv3/vol-comp
network.compression.min-size: 50
network.compression.compression-level: 8
performance.open-behind: off
performance.write-behind: off
performance.strict-write-ordering: on
network.compression.mode: server
network.compression: on
Data is
compressed only
when its size
exceeds the
above value in
bytes.
-1: default
compression (= 8)
0: no compression
1: best speed
9: best
compression
Turn off the
performance
translators
to avoid
Input/output error

# gluster volume start vol-comp
# mount -t glusterfs localhost:/vol-comp /mnt/glusterfs/vol-comp
# dd if=/dev/zero of=/mnt/glusterfs/vol-comp/1gb.dat bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 33.8606 s, 31.7 MB/s
# diff /mnt/glusterfs/vol-comp/1gb.dat /tmp/1gb.dat
#
•  CPU load on client becomes higher than the one without
network compression.
•  Tcpdump showed the 1GB of zero compressed into non-zero
one.
•  High-end CPU might show greater performance.
•  There are still issues and limitations
•  It cannot work with striped volumes.
•  For glusterfs versions <= 3.5, it cannot work with AFR.
117 MB/s when no
compression
Compression and
Decompression
executed correctly

readdir_ahead
Before 3.5.0
volume
read-ahead
Sequential file access can be
fast, but sequential directory
access like "ls" cannot.

readdir_ahead
After 3.5.0
volume
Sequential reads of large
directories can complete
faster!
readdir-aheadread-ahead

readdir_ahead
# gluster volume set vol0 readdir-ahead enable
volume set: success
# gluster volume info vol0
Volume Name: vol0
Type: Distribute
Volume ID: cf9db2aa-5ee8-40c3-8ca9-8316ab31ba59
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
performance.readdir-ahead: enable
disabled by default
How-to

readdir_ahead
# brick="/mnt/lv4/vol4"; gluster volume create vol4 eins:$brick zwei:$brick drei:$brick
vier:$brick fuenf:$brick sechs:$brick
# gluster volume start vol4
# mount -t glusterfs localhost:/vol4 /mnt/glusterfs/vol4
# mkdir /mnt/glusterfs/vol4/manyfiles
# for a in `seq 0 9`; do for b in `seq 0 9`; do for c in `seq 0 9`; for d in `seq 0 9`; do
for e in `seq 0 9`; do for f in `seq 0 9`; do for g in `seq 0 9`; do for h in `seq 0 9`; do
for i in `seq 0 9`; do file="/mnt/glusterfs/vol4/manyfiles/8kb${a}${b}${c}${d}${e}${f}${g}${h}
${i}.dat"; echo ${file}; dd if=/dev/zero of=${file} bs=1K count=8; if [ $? -ne 0 ]; then
break; fi; done; done; done; done; done; done; done; done
...
^C
# df -ki /mnt/glusterfs/vol4
Filesystem Inodes IUsed IFree IUse% Mounted on
localhost:vol4 314572800 3394646 311178154 2% /mnt/glusterfs/vol4
# umount /mnt/glusterfs/vol4
Setup for evaluation
3 million 8K files

readdir_ahead
# for i in `seq 0 2`; do time ls /mnt/glusterfs/vol4/manyfiles > /dev/null; done
26.24s user 18.70s system 6% cpu 12:03.05 total
# umount /mnt/glusterfs/vol4
Evaluation
# gluster volume stop vol4 && gluster volume start vol4
# gluster volume set vol4 readdir-ahead enable
# for i in `seq 0 2`; do time ls /mnt/glusterfs/vol4/manyfiles > /dev/null; done
# gluster volume reset vol4
# umount /mnt/glusterfs/vol4 1.68 times faster!

Stability
Enhancements

Prevent NFS restart on Volume change (Part 1)
Gluster NFS Graph
nfs/server
option nfs3.vol4.volume-id 706122a9-44fc-4d1d-8c3b-97482d98b95c
option rpc-auth.addr.vol4.allow *
option nfs3.vol-gfid-access.volume-id 73abf812-4fff-42bd-822b-3036b72f060d
option rpc-auth.addr.vol-gfid-access.allow *
option nfs3.vol2.volume-id d0517697-5372-44a1-960f-6db0d988f3b2
option nfs3.vol-comp.volume-id 92b47734-2552-4168-b3c3-151093562e4f
option rpc-auth.addr.vol-comp.allow *
option nfs3.vol1.volume-id ba03d1e6-a520-4e7f-ac4c-2440a205e80e
option nfs3.vol0.volume-id cf9db2aa-5ee8-40c3-8ca9-8316ab31ba59
option nfs.drc on
option nfs.nlm on
option nfs.dynamic-volumes on
vol0
debug/io-stats
vol0-write-behind
performance/write-behind
vol0-dht
cluster/distribute
vol0-client-0
protocol/client
vol0-client-1
protocol/client
vol1
debug/io-stats
vol1-write-behind
vol1-dht
cluster/distribute
vol1-client-0
protocol/client
vol1-client-1
protocol/client
vol2
debug/io-stats
vol2-write-behind
vol2-dht
cluster/distribute
vol2-client-0
protocol/client
vol2-client-1
protocol/client
Single nfs/server
exists on the top of
all the volumes

Vol2
My presentation last year (2013)
!  NFS and Multi-tenancy
! 'nfs.rpc-auth-allow' for multi-tenancy
! some operations on a volume affect
IOs to other volumes
Vol1
Vol0
e.g.
gluster volume set ...
IO
IO
IO
Vol2
Vol1
Vol0
IO
IO
IO

!   "Some operations" on a volume
!  gluster volume {set|reset} <volumeName> nfs.rpc-auth-allow
!  gluster volume {start|stop} <volumeName>
!  gluster volume add-brick
!  gluster volume remove-brick <volumeName> <brick1> ... <brickn>
commit

!   Internal NFS options became unaffected by volume changes.
!  nfs.readdir-size
!  nfs.nlm
!  nfs.acl
!  nfs.mount-rmtab
!  nfs.drc
!  nfs.drc-size
!  nfs.read-size
!  nfs.write-size
!  nfs.readdir-size
!  nfs.export-dir
!  nfs.export-dirs
!  nfs.enable-ino32
!  nfs.export-volumes
!  nfs.addr-namelookup
!  nfs.outstanding-rpc-limit
!  nfs.mount-mtab
!  nfs.register-with-portmap

Geo-Replication Enhancement
storage
pool
(a cluster)
gsyncd
Before 3.5.0
SPOF!
identify file
changes with
xattrs
directory crawl
with rsync

storage
pool
(a cluster)
gsyncd
for each
peer
After 3.5.0
identify file changes with
changelog in memory
gsyncd
for each
peer
gsyncd
for each
peer
gsyncd
for each
peer
gsyncd
for each
peer
gsyncd
for each
peer

# cat /var/lib/glusterd/vols/vol0/vol0.eins.mnt-lv0-vol0.vol
volume vol0-posix
type storage/posix
option volume-id cf9db2aa-5ee8-40c3-8ca9-8316ab31ba59
option directory /mnt/lv0/vol0
end-volume
volume vol0-changelog
type features/changelog
option changelog-dir /mnt/lv0/vol0/.glusterfs/changelogs
option changelog-brick /mnt/lv0/vol0
subvolumes vol0-posix
end-volume
...
volume vol0-server
type protocol/server
option auth.addr./mnt/lv0/vol0.allow *
option auth.login.863ccc05-1ba2-47cc-8a15-240ad4e8c736.password c8d200d6-db0b-4f87-
be0f-664e08f4ceee
option auth.login./mnt/lv0/vol0.allow 863ccc05-1ba2-47cc-8a15-240ad4e8c736
option transport-type tcp
subvolumes /mnt/lv0/vol0
end-volume
Changelog

# ls -a /mnt/lv0/vol0/.glusterfs/changelogs
. ..
Changelog (contd.)
No use without
gsyncd???

Security
Enhancement

Disk encryption
Write ops
Read ops
1. open
and
write
2. Encryption
3. Transport
4. Write the
encrypted data
to disk
1. open
and
read
4. Decryption 3. Transport
2. read from
underlying disks

Disk encryption
Volume Name: vol2
Type: Replicate
Volume ID: e0332771-a3c2-4fe5-980c-b3860cfe3baf
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
# gluster volume set vol2 encryption on
volume set: success
# for x in quick-read write-behind open-behind; do gluster volume set vol2 performance.$x
off; done
# gluster volume set vol2 encryption.master-key /var/lib/glusterd/vols/vol2/
encryption.master-key
# openssl rand -hex 32 > /var/lib/glusterd/vols/vol2/encryption.master-key
# gluster volume set vol2 encryption.data-key-size 512
Setup

Disk encryption
Volume Name: vol2
Type: Replicate
Status: Started
Transport-type: tcp
Bricks:
encryption.data-key-size: 512
encryption.master-key: /var/lib/glusterd/vols/vol2/encryption.master-key
performance.quick-read: off
features.encryption: on
# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/var/lib/glusterd/vols/vol2/
encryption.master-key localhost:/vol2 /mnt/glusterfs/vol2
Setup (contd.)

Disk encryption
# echo "test" > /mnt/glusterfs/vol2/test.txt
# cat /mnt/glusterfs/vol2/test.txt
test
[eins]# cat /mnt/lv2/vol2/test.txt
Zd??]K!q??tuv
[zwei]# cat /mnt/lv2/vol2/test.txt
Zd??]K!q??tuv
Encryption test
# dd if=/dev/zero of=/mnt/glusterfs/vol1/test.dat bs=1 count=32
# dd if=/dev/zero of=/mnt/glusterfs/vol2/test.dat bs=1 count=32
[eins]# dd if=/dev/zero of=/tmp/test.dat bs=1 count=32
[eins]# diff /tmp/test.dat /mnt/lv2/vol2/test.dat
Binary files /tmp/test.dat and /mnt/lv2/vol2/test.dat differ
[eins]# diff /tmp/test.dat /mnt/lv1/vol1/test.dat
#
# tcpdump -i eth0 -XX
Can see the
transported
zeroed data
fully encrypted.
ASCII files on the
bricks are
encrypted.
Binary files on the
bricks are also
encrypted.
Confirm that no use
of encryption never
encrypt the data, so
you can access the
raw data on several
bricks without
encryption.

Disk encryption
# dd if=/dev/zero of=/tmp/1gb.dat bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.61505 s, 297 MB/s
# diff3 /tmp/1gb.dat /mnt/glusterfs/vol1/1gb.dat /mnt/glusterfs/vol2/1gb.dat
#
Decryption test
Perfect!

Disk encryption
# dd if=/dev/zero of=/mnt/glusterfs/vol1/1gb.dat bs=1M count=1024
1024+0 records in
1024+0 records out
# dd if=/dev/zero of=/mnt/glusterfs/vol2/1gb.dat bs=1M count=1024
1024+0 records in
1024+0 records out
Performance test

Disk encryption
# mount -t nfs -o vers=3,hard,intr,nosuid localhost:/vol2 /mnt/nfs/vol2
mount.nfs: Connection timed out
Work with NFS? (No!)

Disk encryption
# cp /var/lib/glusterd/vols/vol2/encryption.master-key /tmp
# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/tmp/encryption.master-key
localhost:/vol2 /mnt/glusterfs/vol-crypt
# diff /mnt/glusterfs/vol-crypt/test.txt /tmp/test.txt
#
Compromising with the same MK

Disk encryption
# openssl rand -hex 32 > /tmp/encryption.master-key
#
Compromising with a different MK keeping mounted

Disk encryption
# umount /mnt/glusterfs/vol-crypt
# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/tmp/encryption.master-key
localhost:/vol2 /mnt/glusterfs/vol-crypt
diff: /mnt/glusterfs/vol-crypt/test.txt: Invalid argument
# ls -lh /mnt/glusterfs/vol-crypt
total 1.1G
-rw-r--r-- 1 root root 1.0G May 18 23:31 1gb.dat
-rw-r--r-- 1 root root 32 May 18 22:57 test.dat
-rw-r--r-- 1 root root 5 May 18 22:55 test.txt
# cp /mnt/glusterfs/vol-crypt/test.txt ~/
cp: reading `/mnt/glusterfs/vol-crypt/test.txt': Invalid argument
# ls -l ~/test.txt
-rw-r--r-- 1 root root 0 May 19 00:38 /root/test.txt
Compromising with an invalid MK

Disk encryption
# echo "test2" > /mnt/glusterfs/vol-crypt/test2.txt
# cat /mnt/glusterfs/vol-crypt/test2.txt
test2
# diff /mnt/glusterfs/vol-crypt/test2.txt /tmp/test2.txt
#
Compromising with an invalid MK (contd.)
# rm /mnt/glusterfs/vol-crypt/test.txt
rm: cannot remove `/mnt/glusterfs/vol-crypt/test.txt': Invalid argument
total 1.1G
-rw-r--r-- 1 root root 6 May 19 00:39 test2.txt
# rm /mnt/glusterfs/vol-crypt/test2.txt
total 1.1G
Enable to write
a file with an
invalid MK.
(Is it okay?)

Disk encryption
# mv /mnt/glusterfs/vol-crypt/test.txt /mnt/glusterfs/vol-crypt/test2.txt
mv: cannot move `/mnt/glusterfs/vol-crypt/test.txt' to a subdirectory of itself, `/mnt/
glusterfs/vol-crypt/test2.txt'
Compromising with an invalid MK (contd.)
# umount /mnt/glusterfs/vol-crypt
# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/var/lib/glusterd/vols/vol2/
encryption.master-key localhost:/vol2 /mnt/glusterfs/vol-crypt
total 1.1G
-rw-r--r-- 1 root root 6 May 19 00:44 test2.txt
cat: /mnt/glusterfs/vol-crypt/test2.txt: Invalid argument
# rm /mnt/glusterfs/vol-crypt/test2.txt
rm: cannot remove `/mnt/glusterfs/vol-crypt/test2.txt': Invalid argument
The proper user
cannot handle
the file created
with the invalid
MK.
(Is it okay?)

Disk encryption
# gluster volume info vol-crypt
Volume Name: vol2
Type: Replicate
Status: Started
Transport-type: tcp
Bricks:
encryption.data-key-size: 512
encryption.master-key: /var/lib/glusterd/vols/vol2/encryption.master-key
performance.quick-read: off
features.encryption: on
# gluster volume reset vol-crypt
volume reset: success: reset volume successful
Compromising with volume reset

Disk encryption
# gluster volume info vol-crypt
Volume Name: vol2
Type: Replicate
Status: Started
Transport-type: tcp
Bricks:
Compromising with volume reset (contd.)
U�%U?0��x^-�bO
# cat /mnt/glusterfs/vol-crypt/test.txt
Zd��]K!q�tuv
May be a way
of cracking?

Enhancement
for Developers

A Volume
GFID Access
A Volume/.gfid
62fe0d4f-dfe9-4a2d-b811-176c6d347a7c
52fd93ea-45ba-47f0-916a-bd3774239237
e5593949-79c3-463c-909b-8cc8ef014eb4
bf758c70-ff2b-4f0d-bfc9-860ece79c246
70aaacf9-1c09-44e2-97a2-9486adf10225
e5498fc4-7345-4f5f-af59-81acff1fd083
f6a608ed-0c68-4d1a-a4d7-fb375ba8fd63
d420cbb3-c1e8-47d3-b317-0c8afbc7a8c4
a76dd563-e878-45a0-ac48-59084d86bd0c
f9dbc760-c8f3-41e6-8d24-68b24c4c577b
5c0374a8-18fe-4dd6-89e7-f6551111d980
You can deal
with each file by
GFID
Single
namespace, just
under the mount
point

GFID Access
# brick="/mnt/lv3/vol-gfid-access";gluster volume create vol-gfid-access eins:$brick zwei:
$brick
# gluster volume start vol-gfid-access
# mkdir /mnt/glusterfs/vol-gfid-access
# mount.glusterfs -o aux-gfid-mount localhost:/vol-gfid-access /mnt/glusterfs/vol-gfid-
access
# for i in `seq 0 9`; do dd if=/dev/zero of=/mnt/glusterfs/vol-gfid-access/$i.dat bs=1M
count=1; done
# ls -a /mnt/glusterfs/vol-gfid-access/.gfid
ls: cannot open directory /mnt/glusterfs/vol-gfid-access/.gfid: Stale file handle
# ls -a '/mnt/glusterfs/vol-gfid-access/.gfid/0svu9Cc1wVRLOBiu5NqF3ncw=='
ls: cannot access /mnt/glusterfs/vol-gfid-access/.gfid/0svu9Cc1wVRLOBiu5NqF3ncw==: No such
file or directory
# ls -ld /mnt/glusterfs/vol-gfid-access/.gfid/
drwxr-xr-x 3 root root 166 May 19 03:03 /mnt/glusterfs/vol-gfid-access/.gfid/

GFID Access
# stat /mnt/glusterfs/vol-gfid-access/.gfid/
File: `/mnt/glusterfs/vol-gfid-access/.gfid/'
Size: 166 Blocks: 0 IO Block: 131072 directory
Device: 16h/22d Inode: 13 Links: 3
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2014-05-19 03:03:14.146605880 +0900
Modify: 2014-05-19 03:03:04.968605874 +0900
Change: 2014-05-19 03:03:04.968605874 +0900
# strace ls -a /mnt/glusterfs/vol-gfid-access/.gfid
...
stat("/mnt/glusterfs/vol-gfid-access/.gfid", {st_mode=S_IFDIR|0755, st_size=166, ...}) = 0
open("/mnt/glusterfs/vol-gfid-access/.gfid", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1
ESTALE (Stale file handle)
...
•  How can I let it work well?
•  If it becomes to work fine, applications using GlusterFS can
manage their data in a single namespace.

Additional
news

Added files
!   cli/src/cli-quoted-client
! contrib/qemu
!   error-codes.json
!   extras
!  geo-rep
!  glusterfs-georep-logrotate
!  gluster-rsyslog-*.conf
!  hook-scripts/add-brick
!  logger.conf.example
!  post-upgrade-script-for-quota.sh
!  pre-upgrade-script-for-quota.sh
!   geo-replication
! gf-error-codes.h.template
! libgfchangelog.pc.in
! libglusterfs/src
!  client_t
!  glusterfs-acl
!  timespec
! rpc/rpc-lib/src/rpc-drc
!   run-tests.sh
!   tests
! xlators
!  cluster
!  dht/src
!  dht-shared.c
!  encryption
!  crypt
!  features
!  changelog
!  compress
!  gfid-access
! glupy
! qemu-block
!  quota
!  quota-enforcer-client.c
!  quoted-aggregator
!  quoted-helpers
!  performance
!  readdir-ahead
! playground
!  storage
!  bd (replacement of
bd_map)
related qemu
codes
glupy
has merged!
a lot of test
codes!
template for
xlator
development

Conclusion

Conclusion
!  Update for "everyone"
! 12 features, 8 categories.
!  Contribution by HekaFS
! Disk encryption has been one of my dream since 2.0.2.
!  Voice of users
! Brick Failure Detection
! Prevent NFS restart on Volume change
These are just the great community's power!
Use the latest version, and join us!

To contact us, e-mail here -> storage-contact@nttpc.co.jp

Trying and evaluating the new features of GlusterFS 3.5

More Related Content

What's hot

Similar to Trying and evaluating the new features of GlusterFS 3.5

More from Keisuke Takahashi

Recently uploaded

Trying and evaluating the new features of GlusterFS 3.5