Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

1 
Agenda

!
!
!
!
!

 About me
 Tips 1: Installation
 Tips 2: NFS Workloads
 Tips 3: Logging
 Conclusion

Copyright  (C)  20...
About
Me
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

3 
About me

!  Work at
!  Programmer
! 

as a ...

(Most recently)

! Also Interested in

!  Software Engineer
! Cloud Compu...
About me

! GlusterFS since 2007 (v1.3.7)
!  for my internet crawler at first.

!  Love Gluster because of the ...
!  Pote...
Getting started today!

See http://www.gluster.org/community/documentation/index.php/QuickStart
Copyright  (C)  2013,  NTT...
Tips 1
Installation
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

7 
Tips 1: Installation
!   Question 1: Have you installed GlusterFS?

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  A...
Tips 1: Installation
!   Question 1: Have you installed GlusterFS?
!   Question 2: Did you find it easy?

Copyright  (C)  ...
Tips 1: Installation
!   Question 1: Have you installed GlusterFS?
!   Question 2: Did you find it easy?
!   Question 3: D...
Tips 1: Installation
!
!
!
!

 
 
 
 

Question 1: Have you installed GlusterFS?
Question 2: Did you find it easy?
Questio...
Tips 1: Installation
!
!
!
!
!

 
 
 
 
 

Question 1: Have you installed GlusterFS?
Question 2: Did you find it easy?
Que...
Tips 1: Installation

Development team says that hand-to-hand construction of systems is inefficient so they want to
show ...
Tips 1: Installation

Operational team says that they never want to learn the usage of deployment tools and never
trust th...
Tips 1: Installation

My offer is that what development team should do is only copy the commands that the
deployment tool ...
Tips 1: Installation

Deployment
Automation
Tools
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserv...
Tips 1: Installation
A node for automated deployment
deployment
automation
tool

peer 1

peer 2

sshd

sshd

shell

3

4

...
Tips 1: Installation
A node for automated deployment
deployment
automation
tool

peer 1

peer 2

sshd

sshd

shell

3

4

...
Tips 1: Installation
A node for automated deployment
deployment
automation
tool

peer 1

peer 2

sshd

sshd

shell

3

4

...
Tips 1: Installation
A node for automated deployment
deployment
automation
tool

peer 1

peer 2

sshd

sshd

shell

3

4

...
Tips 1: Installation
A node for automated deployment
deployment
automation
tool

peer 1

peer 2

sshd

sshd

shell

3

4

...
Tips 1: Installation

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

22 
Tips 1: Installation

I use this mainly in my projects
written in Ruby so am used to it.

Copyright  (C)  2013,  NTTPC  Co...
Capistrano

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

24 
Capistrano
$ gem install capistrano
$ cap install
mkdir -p config/deploy
create config/deploy.rb
create config/deploy/stag...
capistrano-glusterfs

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

26 
capistrano-glusterfs
$ tree capistrano-glusterfs
capistrano-glusterfs
├── Capfile
├── Gemfile
├── Gemfile.lock
├── LICENSE...
capistrano-glusterfs
$ tree capistrano-glusterfs
capistrano-glusterfs
├── Capfile
├── Gemfile
├── Gemfile.lock
├── LICENSE...
capistrano-glusterfs
$ cat capistrano-glusterfs/config/deploy/staging.rb
set :stage, :staging
set :branch, "master"
server...
capistrano-glusterfs
$ bundle exec cap -vT | grep -i "glusterfs:"
cap glusterfs:build
# Starting installation
cap glusterf...
capistrano-glusterfs

http://www.capistranorb.com/documentation/getting-started/flow/

Copyright  (C)  2013,  NTTPC  Commu...
capistrano-glusterfs

prepare and install
dependancies
before starting

build GlusterFS from
source and install it
after p...
capistrano-glusterfs
task :preparation do
....
execute "sudo -i yum install -y wget git yum-priorities"
....
end

Copyrigh...
capistrano-glusterfs
task :dependancies do
....
dependancies = %w[
openssh-clients
make
rpm-build
bison
flex
automake
libt...
capistrano-glusterfs
task :compile do
on roles(:glusterfs), in: :sequence, wait: 5 do
execute "cd #{DIR}; sh ./autogen.sh"...
capistrano-glusterfs
task :make_install do
on roles(:glusterfs), in: :sequence, wait: 5 do
execute "cd #{DIR}; sudo make i...
capistrano-glusterfs
$ git clone git@github.com:keithseahus/capistrano-glusterfs.git
$ cd capistrano-glusterfs
$ bundle ex...
capistrano-glusterfs
$ git clone git@github.com:keithseahus/capistrano-glusterfs.git
$ cd capistrano-glusterfs
$ bundle ex...
capistrano-glusterfs
...
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
DEBUG
D...
capistrano-glusterfs
$ bundle exec cap staging gluster:pool:list
INFO [201946fe] Running /usr/bin/env sudo -i gluster pool...
capistrano-glusterfs
$ bundle exec cap staging gluster:volume:help
INFO [4628fd38] Running /usr/bin/env sudo -i gluster vo...
capistrano-glusterfs

capistrano-glusterfs is available here:
https://github.com/keithseahus/capistrano-glusterfs

Copyrig...
Tips 2
NFS Workloads

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

43 
Tips 2: NFS Workloads
!   Question 1: Do you mount GlusterFS in NFS?

Copyright  (C)  2013,  NTTPC  Communications,  Inc. ...
Tips 2: NFS Workloads
!   Question 1: Do you mount GlusterFS in NFS?
!   Question 2: Which is your main workload, write or...
Tips 2: NFS Workloads

peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

fs

...
glusterfsd
brick

brick

br...
Tips 2: NFS Workloads

TCP connection
in gluster protocol
peer 1
glusterd

glustershd

peer 2
glusterfs

d

shd

3

4

N

...
Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs

page
cache
kernel

peer 1
glusterd

glustershd

peer 2
glusterfs

...
Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs

page
cache

TCP connection
in NFS v3

kernel

peer 1
glusterd

glu...
Tips 2: NFS Workloads
NFS Client
read access
to a file

user app
/mnt/nfs
vfs

page
cache
kernel

peer 1
glusterd

gluster...
Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs

page
cache
kernel

performance/io-cache

peer 1
glusterd

glusters...
Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs

activate cachefilesd
/var/cache/fscache

page
cache

cachefilesd

...
Tips 2: NFS Workloads
NFS Client
user app
/mnt/nfs
vfs

read files from its own RAM!

/var/cache/fscache

page
cache

cach...
Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs

Copyright  (C)  2013,  NTTPC  Communications,  ...
Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs
# yum install -y cachefilesd

Copyright  (C)  20...
Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs
# yum install -y cachefilesd
# if test `grep 'SE...
Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs
# yum install -y cachefilesd
# if test `grep 'SE...
Tips 2: NFS Workloads
# NFS_SERVER=192.168.200.101
# MOUNT_POINT=/mnt/nfs
# yum install -y cachefilesd
# if test `grep 'SE...
Tips 2: NFS Workloads
# free -m
root@dev-centos:/root
total
Mem:
996
-/+ buffers/cache:
Swap:
2015

used
926
314
0

free
6...
Tips 2: NFS Workloads
nodes:
VirtualBox
CentOS 6.4 x86_64
4 vCPUs, Core i7 2.4GHz
2GB RAM
Striped SSDs

Copyright  (C)  20...
Tips 2: NFS Workloads
nodes:
VirtualBox
CentOS 6.4 x86_64
4 vCPUs, Core i7 2.4GHz
2GB RAM
Striped SSDs
postmark 1.51
set s...
Tips 2: NFS Workloads
nodes:
VirtualBox
CentOS 6.4 x86_64
4 vCPUs, Core i7 2.4GHz
2GB RAM
Striped SSDs
MB/sec (read)
postm...
Tips 2: NFS Workloads

Okay... but this is still a hypothesis.
We need your feedback!

Copyright  (C)  2013,  NTTPC  Commu...
Tips 3
Logging

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

64 
Tips 3: Logging
!   Question 1: Do you monitor GlusterFS logs?

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  ...
Tips 3: Logging
!   Question 1: Do you monitor GlusterFS logs?
!   Question 2: Do you utilize the logs?

Copyright  (C)  2...
Tips 3: Logging
!   Question 1: Do you monitor GlusterFS logs?
!   Question 2: Do you utilize the logs?
!   Question 3: Do...
Tips 3: Logging

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

68 
Tips 3: Logging

I pick it up because:
•  easy to install
•  easy to configure
•  make unstructured logs structured
•  col...
Tips 3: Logging
NFS Client
user app
/mnt/nfs
vfs

/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1
glusterd

gl...
Tips 3: Logging
NFS Client

Log server

user app
/mnt/nfs
vfs

/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1...
Tips 3: Logging
NFS Client

Log server

user app
/mnt/nfs
vfs

/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1...
Tips 3: Logging
NFS Client

Log server

user app
/mnt/nfs
vfs

/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1...
Tips 3: Logging
NFS Client

Log server

user app
/mnt/nfs
vfs

/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1...
Tips 3: Logging
NFS Client

Log server

user app
/mnt/nfs
vfs

/var/cache/fscache

page
cache

cachefilesd

kernel

peer 1...
Tips 3: Logging
/etc/td-agent/td-agent.conf (on the log server)
<source>
type forward
port 24224
bind 0.0.0.0
</source>
<m...
Tips 3: Logging
/etc/td-agent/td-agent.conf (on the GlusterFS peers)
<source>
type tail
path /var/log/glusterfs/usr-local-...
Tips 3: Logging

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

78 
Tips 3: Logging

You can specify log levels
to forward.

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights ...
Tips 3: Logging

You can specify each field
name to output in JSON.

Comment out if you don't
need the fields.

Copyright ...
Tips 3: Logging

Parse error should be
treated as a failure of
fluent-plugin-glusterfs.

Copyright  (C)  2013,  NTTPC  Com...
Tips 3: Logging
It makes the logs structured in JSON.
[root@glusterfs-unstable-01 plugin]# tail -3 /var/log/glusterfs/usr-...
Tips 3: Logging
The logs on each peer could be output in a single log file.
[root@dev-centos ~]# grep glusterfs-unstable-0...
Tips 3: Logging

Also available on Gluster Forge:
https://forge.gluster.org/fluent-plugin-glusterfs

Copyright  (C)  2013,...
Tips 3: Logging

More detailed explanation in Japanese is here!:
http://www.nttpc.co.jp/company/r_and_d/technology/gluster...
Conclusion

Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.  

86 
Conclusion

!  Deployment automation tools are
useful to install and operate
GlusterFS on multiple nodes.
!  With cachefil...
To contact us, e-mail here -> storage-contact@nttpc.co.jp
Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights...
Upcoming SlideShare
Loading in …5
×

Gluster Cloud Night in Tokyo 2013 -- Tips for getting started

7,030 views

Published on

My presentation on Nov. 13th, 2013.

Published in: Technology, Business
  • Be the first to comment

Gluster Cloud Night in Tokyo 2013 -- Tips for getting started

  1. 1. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   1 
  2. 2. Agenda ! ! ! ! !  About me  Tips 1: Installation  Tips 2: NFS Workloads  Tips 3: Logging  Conclusion Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   2 
  3. 3. About Me Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   3 
  4. 4. About me !  Work at !  Programmer !  as a ... (Most recently) ! Also Interested in !  Software Engineer ! Cloud Computing ! Big Data ! Something new technologies ! Supporting GlusterFS/Red Hat Storage Introduction with Red Hat K.K. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   4 
  5. 5. About me ! GlusterFS since 2007 (v1.3.7) !  for my internet crawler at first. !  Love Gluster because of the ... !  Potential !  Performance !  Code !  Community !  Introduced or introducing it into ... !  Printer and scanner solution (field trial) !  Email services !  File storage services (WebDAV, NFS) !  Backup services !  Shared storage platform !  A board member of the Gluster Community Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   5 
  6. 6. Getting started today! See http://www.gluster.org/community/documentation/index.php/QuickStart Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   6 
  7. 7. Tips 1 Installation Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   7 
  8. 8. Tips 1: Installation !   Question 1: Have you installed GlusterFS? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   8 
  9. 9. Tips 1: Installation !   Question 1: Have you installed GlusterFS? !   Question 2: Did you find it easy? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   9 
  10. 10. Tips 1: Installation !   Question 1: Have you installed GlusterFS? !   Question 2: Did you find it easy? !   Question 3: Did you do it on each single node? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   10 
  11. 11. Tips 1: Installation ! ! ! !         Question 1: Have you installed GlusterFS? Question 2: Did you find it easy? Question 3: Did you do it on each single node? Question 4: Don't you think you should take easier way? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   11 
  12. 12. Tips 1: Installation ! ! ! ! !           Question 1: Have you installed GlusterFS? Question 2: Did you find it easy? Question 3: Did you do it on each single node? Question 4: Don't you think you should take easier way? Question 5: Do you use deployment automation tools? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   12 
  13. 13. Tips 1: Installation Development team says that hand-to-hand construction of systems is inefficient so they want to show an installation way with a deployment tool in the documents for installation. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   13 
  14. 14. Tips 1: Installation Operational team says that they never want to learn the usage of deployment tools and never trust them so they don't accept such a document and they will install the system by hand work. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   14 
  15. 15. Tips 1: Installation My offer is that what development team should do is only copy the commands that the deployment tool write on your stdout and paste them into the documents for operational team. https://twitter.com/keithseahus/status/398453753352691714 Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   15 
  16. 16. Tips 1: Installation Deployment Automation Tools Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   16 
  17. 17. Tips 1: Installation A node for automated deployment deployment automation tool peer 1 peer 2 sshd sshd shell 3 4 N shell ... Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   17 
  18. 18. Tips 1: Installation A node for automated deployment deployment automation tool peer 1 peer 2 sshd sshd shell 3 4 N shell ... Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   18 
  19. 19. Tips 1: Installation A node for automated deployment deployment automation tool peer 1 peer 2 sshd sshd shell 3 4 N shell ... glusterfs glusterfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   19 
  20. 20. Tips 1: Installation A node for automated deployment deployment automation tool peer 1 peer 2 sshd sshd shell 3 4 N shell ... glusterfs glusterfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   20 
  21. 21. Tips 1: Installation A node for automated deployment deployment automation tool peer 1 peer 2 sshd sshd shell 3 4 N shell ... glusterfs glusterfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   21 
  22. 22. Tips 1: Installation Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   22 
  23. 23. Tips 1: Installation I use this mainly in my projects written in Ruby so am used to it. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   23 
  24. 24. Capistrano Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   24 
  25. 25. Capistrano $ gem install capistrano $ cap install mkdir -p config/deploy create config/deploy.rb create config/deploy/staging.rb create config/deploy/production.rb mkdir -p lib/capistrano/tasks Capified $ tree . . ├── Capfile ├── config │   ├── deploy │   │   ├── production.rb │   │   └── staging.rb │   └── deploy.rb └── lib └── capistrano └── tasks Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   25 
  26. 26. capistrano-glusterfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   26 
  27. 27. capistrano-glusterfs $ tree capistrano-glusterfs capistrano-glusterfs ├── Capfile ├── Gemfile ├── Gemfile.lock ├── LICENSE ├── README.md ├── config │   ├── deploy │   │   ├── production.rb │   │   ├── production.rb.example │   │   ├── staging.rb │   │   └── staging.rb.example │   ├── deploy.rb │   └── deploy.rb.example └── lib └── capistrano └── tasks ├── gluster.cap ├── glusterd.cap └── glusterfs.cap Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   27 
  28. 28. capistrano-glusterfs $ tree capistrano-glusterfs capistrano-glusterfs ├── Capfile ├── Gemfile ├── Gemfile.lock ├── LICENSE ├── README.md ├── config │   ├── deploy │   │   ├── production.rb ... settings for production environment │   │   ├── production.rb.example │   │   ├── staging.rb ... settings for staging environment │   │   └── staging.rb.example │   ├── deploy.rb ... global settings and main sequences │   └── deploy.rb.example └── lib └── capistrano └── tasks ├── gluster.cap ... tasks to execute gluster command remotely ├── glusterd.cap ... tasks to start/stop all remote glusterd └── glusterfs.cap ... tasks for deploying GlusterFS Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   28 
  29. 29. capistrano-glusterfs $ cat capistrano-glusterfs/config/deploy/staging.rb set :stage, :staging set :branch, "master" server server server server server server server server server server '192.168.200.100', '192.168.200.101', '192.168.200.102', '192.168.200.103', '192.168.200.104', '192.168.200.105', '192.168.200.106', '192.168.200.107', '192.168.200.108', '192.168.200.109', roles: roles: roles: roles: roles: roles: roles: roles: roles: roles: %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs %w(glusterfs glusterd gluster), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' glusterd), user: 'deploy' Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   29 
  30. 30. capistrano-glusterfs $ bundle exec cap -vT | grep -i "glusterfs:" cap glusterfs:build # Starting installation cap glusterfs:compile # Compile GlusterFS cap glusterfs:configuration # GlusterFS related configurations cap glusterfs:dependancies # Install GlusterFS dependancies cap glusterfs:make_install # Install GlusterFS cap glusterfs:preparation # Prepare installing GlusterFS $ bundle exec cap -vT | grep -i "glusterd:" cap glusterd:restart # Restart glusterd cap glusterd:start # Start glusterd cap glusterd:status # Get the status of glusterd cap glusterd:stop # Stop glusterd $ bundle exec cap -vT | grep -i "gluster:" cap gluster:peer:help # Help command for peer cap gluster:peer:status # list status of peers cap gluster:pool:list # list all the nodes in the pool cap gluster:volume:help # display help for the volume command cap gluster:volume:info # list information of all volumes cap gluster:volume:list # list all volumes in cluster cap gluster:volume:status # display status of all volume(s)/brick Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   30 
  31. 31. capistrano-glusterfs http://www.capistranorb.com/documentation/getting-started/flow/ Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   31 
  32. 32. capistrano-glusterfs prepare and install dependancies before starting build GlusterFS from source and install it after published http://www.capistranorb.com/documentation/getting-started/flow/ Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   32 
  33. 33. capistrano-glusterfs task :preparation do .... execute "sudo -i yum install -y wget git yum-priorities" .... end Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   33 
  34. 34. capistrano-glusterfs task :dependancies do .... dependancies = %w[ openssh-clients make rpm-build bison flex automake libtool ncurses-devel readline-devel openssl-devel libxml2-devel libibverbs-devel libacl-devel libattr-devel python-devel python-setuptools lvm2-devel systemtap-sdt-devel libaio-devel xfsprogs ] Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   34 
  35. 35. capistrano-glusterfs task :compile do on roles(:glusterfs), in: :sequence, wait: 5 do execute "cd #{DIR}; sh ./autogen.sh" execute "cd #{DIR}; sh ./configure --prefix=#{INSTALL_PATH} --enable-bd-xlator --enable-fusermount --enable-systemtap --enable-debug" execute "cd #{DIR}; make" end end Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   35 
  36. 36. capistrano-glusterfs task :make_install do on roles(:glusterfs), in: :sequence, wait: 5 do execute "cd #{DIR}; sudo make install" symlink = "/usr/local/glusterfs" execute "sudo -i rm -f #{symlink}" if test("[ -L #{symlink} ]") execute "sudo -i ln -snf #{INSTALL_PATH} #{symlink}" end end Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   36 
  37. 37. capistrano-glusterfs $ git clone git@github.com:keithseahus/capistrano-glusterfs.git $ cd capistrano-glusterfs $ bundle exec cap staging deploy Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   37 
  38. 38. capistrano-glusterfs $ git clone git@github.com:keithseahus/capistrano-glusterfs.git $ cd capistrano-glusterfs $ bundle exec cap staging deploy DEBUG [e5bdd4a7] Running /usr/bin/env /bin/egrep '/bin' ~/.bashrc > /dev/null; exit $? on 192.168.200.101 DEBUG [e5bdd4a7] Command: /bin/egrep '/bin' ~/.bashrc > /dev/null; exit $? DEBUG [e5bdd4a7] Finished in 0.096 seconds with exit status 0 (successful). INFO [a54cf65d] Running /usr/bin/env sudo -i yum install -y wget git yumpriorities on 192.168.200.101 DEBUG [a54cf65d] Command: sudo -i yum install -y wget git yum-priorities DEBUG [a54cf65d] Loaded plugins: fastestmirror, priorities DEBUG [a54cf65d] Loading mirror speeds from cached hostfile epel/metalink | 6.4 kB 00:00 DEBUG [a54cf65d] * base: ftp.tsukuba.wide.ad.jp DEBUG [a54cf65d] * epel: ftp.kddilabs.jp DEBUG [a54cf65d] * extras: ftp.tsukuba.wide.ad.jp DEBUG [a54cf65d] * remi: rpms.famillecollet.com DEBUG [a54cf65d] * remi-php55: rpms.famillecollet.com DEBUG [a54cf65d] * remi-test: rpms.famillecollet.com DEBUG [a54cf65d] * rpmforge: ftp.kddilabs.jp DEBUG [a54cf65d] * updates: mirrors.tuna.tsinghua.edu.cn ... Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   38 
  39. 39. capistrano-glusterfs ... DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG DEBUG ... INFO [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] [80062be9] GlusterFS configure summary =========================== FUSE client : yes Infiniband verbs : no epoll IO multiplex : yes argp-standalone : no fusermount : yes readline : yes georeplication : yes Linux-AIO : yes Enable Debug : yes systemtap : yes Block Device backend : yes glupy : yes Use syslog : yes XML output : yes QEMU Block formats : no [a9e8de7e] Finished in 0.015 seconds with exit status 0 (successful). Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   39 
  40. 40. capistrano-glusterfs $ bundle exec cap staging gluster:pool:list INFO [201946fe] Running /usr/bin/env sudo -i gluster pool list on 192.168.200.101 DEBUG [201946fe] Command: sudo -i gluster pool list DEBUG [201946fe] UUID Hostname State DEBUG [201946fe] d185fa19-d1a1-489c-a653-4510122f5278 localhostConnected INFO [201946fe] Finished in 0.294 seconds with exit status 0 (successful). $ bundle exec cap staging glusterd:status INFO [cb482772] Running /usr/bin/env sudo -i /sbin/service glusterd status on 192.168.200.101 DEBUG [cb482772] Command: sudo -i /sbin/service glusterd status INFO [cb482772] Finished in 0.151 seconds with exit status 0 (successful). DEBUG [cb482772] glusterd (pid 25855) is running... INFO [cb482772] Finished in 0.151 seconds with exit status 0 (successful). Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   40 
  41. 41. capistrano-glusterfs $ bundle exec cap staging gluster:volume:help INFO [4628fd38] Running /usr/bin/env sudo -i gluster volume help on 192.168.200.101 DEBUG [4628fd38] Command: sudo -i gluster volume help DEBUG [4628fd38] volume info [all|<VOLNAME>] - list information of all volumes DEBUG [4628fd38] volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT>] [device vg] [transport <tcp|rdma|tcp,rdma>] <NEW-BRICK> ... [force] - create a new volume of specified type with mentioned bricks DEBUG [4628fd38] volume delete <VOLNAME> - delete volume specified by <VOLNAME> DEBUG [4628fd38] volume start <VOLNAME> [force] - start volume specified by <VOLNAME> DEBUG [4628fd38] volume stop <VOLNAME> [force] - stop volume specified by <VOLNAME> DEBUG [4628fd38] volume add-brick <VOLNAME> [<stripe|replica> <COUNT>] <NEWBRICK> ... [force] - add brick to volume <VOLNAME> DEBUG [4628fd38] volume remove-brick <VOLNAME> [replica <COUNT>] <BRICK> ... {start|stop|status|commit|force} - remove brick from volume <VOLNAME> DEBUG [4628fd38] volume rebalance <VOLNAME> [fix-layout] {start|stop|status} [force] - rebalance operations DEBUG [4628fd38] volume replace-brick <VOLNAME> <BRICK> <NEW-BRICK> {start [force]|pause|abort|status|commit [force]} - replace-brick operations DEBUG [4628fd38] volume set <VOLNAME> <KEY> <VALUE> - set options for volume <VOLNAME> ... Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   41 
  42. 42. capistrano-glusterfs capistrano-glusterfs is available here: https://github.com/keithseahus/capistrano-glusterfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   42 
  43. 43. Tips 2 NFS Workloads Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   43 
  44. 44. Tips 2: NFS Workloads !   Question 1: Do you mount GlusterFS in NFS? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   44 
  45. 45. Tips 2: NFS Workloads !   Question 1: Do you mount GlusterFS in NFS? !   Question 2: Which is your main workload, write or read? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   45 
  46. 46. Tips 2: NFS Workloads peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   46 
  47. 47. Tips 2: NFS Workloads TCP connection in gluster protocol peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   47 
  48. 48. Tips 2: NFS Workloads NFS Client user app /mnt/nfs vfs page cache kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   48 
  49. 49. Tips 2: NFS Workloads NFS Client user app /mnt/nfs vfs page cache TCP connection in NFS v3 kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   49 
  50. 50. Tips 2: NFS Workloads NFS Client read access to a file user app /mnt/nfs vfs page cache kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   50 
  51. 51. Tips 2: NFS Workloads NFS Client user app /mnt/nfs vfs page cache kernel performance/io-cache peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   51 
  52. 52. Tips 2: NFS Workloads NFS Client user app /mnt/nfs vfs activate cachefilesd /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   52 
  53. 53. Tips 2: NFS Workloads NFS Client user app /mnt/nfs vfs read files from its own RAM! /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   53 
  54. 54. Tips 2: NFS Workloads # NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   54 
  55. 55. Tips 2: NFS Workloads # NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   55 
  56. 56. Tips 2: NFS Workloads # NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd # if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne 0; then sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf fi Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   56 
  57. 57. Tips 2: NFS Workloads # NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd # if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne 0; then sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf fi # chkconfig cachefilesd on # service cachefilesd start Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   57 
  58. 58. Tips 2: NFS Workloads # NFS_SERVER=192.168.200.101 # MOUNT_POINT=/mnt/nfs # yum install -y cachefilesd # if test `grep 'SELINUX=disabled' /etc/selinux/config > /dev/null; echo $?` -ne 0; then sed -i 's/^secctx/#secctx/g' /etc/cachefilesd.conf fi # chkconfig cachefilesd on # service cachefilesd start # mount -t nfs -o fsc,vers=3 $NFS_SERVER $MOUNT_POINT Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   58 
  59. 59. Tips 2: NFS Workloads # free -m root@dev-centos:/root total Mem: 996 -/+ buffers/cache: Swap: 2015 used 926 314 0 free 69 682 2015 shared 0 buffers 32 cached 579 # ls /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/ J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040 000g000000000/ @01 @0d @13 @1b @22 @35 @40 @50 @58 @61 @6e @78 @87 @a7 @b3 @b9 @d1 @d7 @e6 @06 @10 @17 @1e @25 @3c @4d @52 @5d @63 @6f @81 @97 @a9 @b6 @c1 @d3 @dc @f1 @0a @11 @1a @21 @2b @3d @4f @57 @5f @66 @73 @82 @9a @ae @b7 @cc @d4 @de @fe # du -h /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/ J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040 000g000000000/ 186M /var/cache/fscache/cache/@4a/I03nfs/@7f/Jc0000000000000cG0I00/@8d/ J1100000000000QFi24tY1EH900000000000M000000g00000100Kb000wFe000jt000oG3000000040 000g000000000/ Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   59 
  60. 60. Tips 2: NFS Workloads nodes: VirtualBox CentOS 6.4 x86_64 4 vCPUs, Core i7 2.4GHz 2GB RAM Striped SSDs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   60 
  61. 61. Tips 2: NFS Workloads nodes: VirtualBox CentOS 6.4 x86_64 4 vCPUs, Core i7 2.4GHz 2GB RAM Striped SSDs postmark 1.51 set size 1048576 10485760 set number 500 set transactions 1000 set location /mnt/nfs/rep2 set subdirectories 100 set read 8192 set write 8192 set buffering true Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   61 
  62. 62. Tips 2: NFS Workloads nodes: VirtualBox CentOS 6.4 x86_64 4 vCPUs, Core i7 2.4GHz 2GB RAM Striped SSDs MB/sec (read) postmark 1.51 set size 1048576 10485760 set number 500 set transactions 1000 set location /mnt/nfs/rep2 set subdirectories 100 set read 8192 set write 8192 set buffering true ↓Oops... Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   62 
  63. 63. Tips 2: NFS Workloads Okay... but this is still a hypothesis. We need your feedback! Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   63 
  64. 64. Tips 3 Logging Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   64 
  65. 65. Tips 3: Logging !   Question 1: Do you monitor GlusterFS logs? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   65 
  66. 66. Tips 3: Logging !   Question 1: Do you monitor GlusterFS logs? !   Question 2: Do you utilize the logs? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   66 
  67. 67. Tips 3: Logging !   Question 1: Do you monitor GlusterFS logs? !   Question 2: Do you utilize the logs? !   Question 3: Do you use a log collector or aggregator? Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   67 
  68. 68. Tips 3: Logging Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   68 
  69. 69. Tips 3: Logging I pick it up because: •  easy to install •  easy to configure •  make unstructured logs structured •  collect logs from multiple resources •  output aggregated and structured logs into a target as you prefer •  easy to create a module •  high performance •  well maintained Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   69 
  70. 70. Tips 3: Logging NFS Client user app /mnt/nfs vfs /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   70 
  71. 71. Tips 3: Logging NFS Client Log server user app /mnt/nfs vfs /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   71 
  72. 72. Tips 3: Logging NFS Client Log server user app /mnt/nfs vfs /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   72 
  73. 73. Tips 3: Logging NFS Client Log server user app /mnt/nfs vfs /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   73 
  74. 74. Tips 3: Logging NFS Client Log server user app /mnt/nfs vfs /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   74 
  75. 75. Tips 3: Logging NFS Client Log server user app /mnt/nfs vfs /var/cache/fscache page cache cachefilesd kernel peer 1 glusterd glustershd peer 2 glusterfs d shd 3 4 N fs ... glusterfsd brick brick brick fsd brick b b b b Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   75 
  76. 76. Tips 3: Logging /etc/td-agent/td-agent.conf (on the log server) <source> type forward port 24224 bind 0.0.0.0 </source> <match filtered.glusterd.**> type file path /home/keith/var/log/fluent/glusterd </match> Today I will get the td-agent output the logs into a file. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   76 
  77. 77. Tips 3: Logging /etc/td-agent/td-agent.conf (on the GlusterFS peers) <source> type tail path /var/log/glusterfs/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log pos_file /var/log/td-agent/usr-local-glusterfs-3.4.0beta2-etc-glusterfs-glusterd.vol.log.pos tag glusterd format /^(?:[(?<date_ymd>[0-9]{4}-[01][0-9]-[0-3][0-9]) (?<time_hms>[0-2][0-9]:[0-5][0-9]:[0-6] [0-9]).(?<time_usec>[0-9]{6})]) (?<log_level>[TDINWECA]) (?:[(?<source_file_name>[^[]:/ ]*):(? <source_line>[0-9]*):(?<function_name>[^[]:/ ]*)]) (?<component>[^[]:/ ]*): *(?<message>.*)$/ rotate_wait 5 </source> <match glusterd.**> type hostname key_name host add_prefix filtered </match> Too complex to input and maintain! <match filtered.glusterd.**> type forward send_timeout 60s recover_wait 10s heartbeat_interval 1s phi_threshold 8 hard_timeout 60s <server> … Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   77 
  78. 78. Tips 3: Logging Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   78 
  79. 79. Tips 3: Logging You can specify log levels to forward. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   79 
  80. 80. Tips 3: Logging You can specify each field name to output in JSON. Comment out if you don't need the fields. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   80 
  81. 81. Tips 3: Logging Parse error should be treated as a failure of fluent-plugin-glusterfs. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   81 
  82. 82. Tips 3: Logging It makes the logs structured in JSON. [root@glusterfs-unstable-01 plugin]# tail -3 /var/log/glusterfs/usr-local-glusterfs-3.4.0beta2-etcglusterfs-glusterd.vol.log [2013-07-07 15:52:43.778961] I [socket.c:3495:socket_init] 0-management: using system polling thread [2013-07-07 15:52:43.779397] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available) [2013-07-07 15:52:43.779668] I [socket.c:2236:socket_event_handler] 0-transport: disconnecting now [root@dev-centos ~]# grep glusterfs-unstable-01 /home/keith/var/log/fluent/glusterd. 20130708.b4e0ed6e32655d35f | tail -3 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"778961","log_level":"I","source_file_name": "socket.c","source_line":"3495","function_name":"socket_init","component":"0management","message":"using system polling thread","host":"glusterfs-unstable-01.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779397","log_level":"W","source_file_name": "socket.c","source_line":"514","function_name":"__socket_rwv","component":"0management","message":"readv failed (No data available)","host":"glusterfs-unstable-01.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779668","log_level":"I","source_file_name": "socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0transport","message":"disconnecting now","host":"glusterfs-unstable-01.localdomain"} Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   82 
  83. 83. Tips 3: Logging The logs on each peer could be output in a single log file. [root@dev-centos ~]# grep glusterfs-unstable-01 /home/keith/var/log/fluent/glusterd. 20130708.b4e0ed6e32655d35f | tail -3 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"778961","log_level":"I","source_file_name": "socket.c","source_line":"3495","function_name":"socket_init","component":"0management","message":"using system polling thread","host":"glusterfs-unstable-01.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779397","log_level":"W","source_file_name": "socket.c","source_line":"514","function_name":"__socket_rwv","component":"0management","message":"readv failed (No data available)","host":"glusterfs-unstable-01.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"779668","log_level":"I","source_file_name": "socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0transport","message":"disconnecting now","host":"glusterfs-unstable-01.localdomain"} [root@dev-centos ~]# grep glusterfs-unstable-02 /home/keith/var/log/fluent/glusterd. 20130708.b4e0ed6e32655d35f | tail -3 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"759502","log_level":"I","source_file_name": "socket.c","source_line":"3495","function_name":"socket_init","component":"0management","message":"using system polling thread","host":"glusterfs-unstable-02.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"759838","log_level":"W","source_file_name": "socket.c","source_line":"514","function_name":"__socket_rwv","component":"0management","message":"readv failed (No data available)","host":"glusterfs-unstable-02.localdomain"} 2013-07-08T00:52:43+09:00 filtered.glusterd {"date_ymd":"2013-07-07","time_hms":"15:52:43","time_usec":"760057","log_level":"I","source_file_name": "socket.c","source_line":"2236","function_name":"socket_event_handler","component":"0transport","message":"disconnecting now","host":"glusterfs-unstable-02.localdomain"} Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   83 
  84. 84. Tips 3: Logging Also available on Gluster Forge: https://forge.gluster.org/fluent-plugin-glusterfs Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   84 
  85. 85. Tips 3: Logging More detailed explanation in Japanese is here!: http://www.nttpc.co.jp/company/r_and_d/technology/gluster_fs_fluentd.html Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   85 
  86. 86. Conclusion Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   86 
  87. 87. Conclusion !  Deployment automation tools are useful to install and operate GlusterFS on multiple nodes. !  With cachefilesd, NFS can might be more efficient under heavy read workload. !  Aggregating GlusterFS logs is easy enough and you can utilize them. Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   87 
  88. 88. To contact us, e-mail here -> storage-contact@nttpc.co.jp Copyright  (C)  2013,  NTTPC  Communications,  Inc.  All  Rights  Reserved.   88 

×