My presentation in LinuxCon/CloudOpen Japan 2014.
It has passed few days since GlusterFS 3.5 released so feel free to correct me if you find my mistakes or misunderstandings. Thanks.
This session will cover performance-related developments in Red Hat Gluster Storage 3 and share best practices for testing, sizing, configuration, and tuning.
Join us to learn about:
Current features in Red Hat Gluster Storage, including 3-way replication, JBOD support, and thin-provisioning.
Features that are in development, including network file system (NFS) support with Ganesha, erasure coding, and cache tiering.
New performance enhancements related to the area of remote directory memory access (RDMA), small-file performance, FUSE caching, and solid state disks (SSD) readiness.
Red Hat Storage Server Administration Deep DiveRed_Hat_Storage
"In this session for administrators of all skill levels, you’ll get a deep technical dive into Red Hat Storage Server and GlusterFS administration.
We’ll start with the basics of what scale-out storage is, and learn about the unique implementation of Red Hat Storage Server and its advantages over legacy and competing technologies. From the basic knowledge and design principles, we’ll move to a live start-to-finish demonstration. Your experience will include:
Building a cluster.
Allocating resources.
Creating and modifying volumes of different types.
Accessing data via multiple client protocols.
A resiliency demonstration.
Expanding and contracting volumes.
Implementing directory quotas.
Recovering from and preventing split-brain.
Asynchronous parallel geo-replication.
Behind-the-curtain views of configuration files and logs.
Extended attributes used by GlusterFS.
Performance tuning basics.
New and upcoming feature demonstrations.
Those new to the scale-out product will leave this session with the knowledge and confidence to set up their first Red Hat Storage Server environment. Experienced administrators will sharpen their skills and gain insights into the newest features. IT executives and managers will gain a valuable overview to help fuel the drive for next-generation infrastructures."
This session will cover performance-related developments in Red Hat Gluster Storage 3 and share best practices for testing, sizing, configuration, and tuning.
Join us to learn about:
Current features in Red Hat Gluster Storage, including 3-way replication, JBOD support, and thin-provisioning.
Features that are in development, including network file system (NFS) support with Ganesha, erasure coding, and cache tiering.
New performance enhancements related to the area of remote directory memory access (RDMA), small-file performance, FUSE caching, and solid state disks (SSD) readiness.
Red Hat Storage Server Administration Deep DiveRed_Hat_Storage
"In this session for administrators of all skill levels, you’ll get a deep technical dive into Red Hat Storage Server and GlusterFS administration.
We’ll start with the basics of what scale-out storage is, and learn about the unique implementation of Red Hat Storage Server and its advantages over legacy and competing technologies. From the basic knowledge and design principles, we’ll move to a live start-to-finish demonstration. Your experience will include:
Building a cluster.
Allocating resources.
Creating and modifying volumes of different types.
Accessing data via multiple client protocols.
A resiliency demonstration.
Expanding and contracting volumes.
Implementing directory quotas.
Recovering from and preventing split-brain.
Asynchronous parallel geo-replication.
Behind-the-curtain views of configuration files and logs.
Extended attributes used by GlusterFS.
Performance tuning basics.
New and upcoming feature demonstrations.
Those new to the scale-out product will leave this session with the knowledge and confidence to set up their first Red Hat Storage Server environment. Experienced administrators will sharpen their skills and gain insights into the newest features. IT executives and managers will gain a valuable overview to help fuel the drive for next-generation infrastructures."
Presentation from 2016 Austin OpenStack Summit.
The Ceph upstream community is declaring CephFS stable for the first time in the recent Jewel release, but that declaration comes with caveats: while we have filesystem repair tools and a horizontally scalable POSIX filesystem, we have default-disabled exciting features like horizontally-scalable metadata servers and snapshots. This talk will present exactly what features you can expect to see, what's blocking the inclusion of other features, and what you as a user can expect and can contribute by deploying or testing CephFS.
Gluster for Geeks: Performance Tuning Tips & TricksGlusterFS
In this Gluster for Geeks technical webinar, Jacob Shucart, Senior Systems Engineer, will provide useful tips and tricks to make a Gluster cluster meet your performance requirements. He will review considerations for all different phases including planning, configuration, implementation, tuning, and benchmarking.
Topics covered will include:
• Protocols (CIFS, NFS, GlusterFS)
• Hardware configuration
• Tuning parameters
• Performance benchmarks
Ariel Waizel discusses the Data Plane Development Kit (DPDK), an API for developing fast packet processing code in user space.
* Who needs this library? Why bypass the kernel?
* How does it work?
* How good is it? What are the benchmarks?
* Pros and cons
Ariel worked on kernel development at the IDF, Ben Gurion University, and several companies. He is interested in networking, security, machine learning, and basically everything except UI development. Currently a Solution Architect at ConteXtream (an HPE company), which specializes in SDN solutions for the telecom industry.
Chicago Docker Meetup Presentation - MediaflyMediafly
Bryan Murphy's presentation from the 2nd Chicago Docker meetup on March 12, 2014 at Mediafly HQ. In his presentation, Bryan explains how we use Docker right now at Mediafly in production.
(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per SecondAmazon Web Services
With the introduction of Amazon Elastic Block Store (EBS) GP2 and recent stability improvements, EBS has gained credibility in the Cassandra world for high performance workloads. By running Cassandra on Amazon EBS, you can run denser, cheaper Cassandra clusters with just as much availability as ephemeral storage instances. This talk walks through a highly detailed use case and configuration guide for a multi PetaByte, million write per second cluster that needs to be high performing and cost efficient. We explore the instance type choices, configuration, and low-level tuning that allowed us to hit 1.3 million writes per second with a replication factor of 3 on just 60 nodes.
Presentation from 2016 Austin OpenStack Summit.
The Ceph upstream community is declaring CephFS stable for the first time in the recent Jewel release, but that declaration comes with caveats: while we have filesystem repair tools and a horizontally scalable POSIX filesystem, we have default-disabled exciting features like horizontally-scalable metadata servers and snapshots. This talk will present exactly what features you can expect to see, what's blocking the inclusion of other features, and what you as a user can expect and can contribute by deploying or testing CephFS.
Gluster for Geeks: Performance Tuning Tips & TricksGlusterFS
In this Gluster for Geeks technical webinar, Jacob Shucart, Senior Systems Engineer, will provide useful tips and tricks to make a Gluster cluster meet your performance requirements. He will review considerations for all different phases including planning, configuration, implementation, tuning, and benchmarking.
Topics covered will include:
• Protocols (CIFS, NFS, GlusterFS)
• Hardware configuration
• Tuning parameters
• Performance benchmarks
Ariel Waizel discusses the Data Plane Development Kit (DPDK), an API for developing fast packet processing code in user space.
* Who needs this library? Why bypass the kernel?
* How does it work?
* How good is it? What are the benchmarks?
* Pros and cons
Ariel worked on kernel development at the IDF, Ben Gurion University, and several companies. He is interested in networking, security, machine learning, and basically everything except UI development. Currently a Solution Architect at ConteXtream (an HPE company), which specializes in SDN solutions for the telecom industry.
Chicago Docker Meetup Presentation - MediaflyMediafly
Bryan Murphy's presentation from the 2nd Chicago Docker meetup on March 12, 2014 at Mediafly HQ. In his presentation, Bryan explains how we use Docker right now at Mediafly in production.
(BDT323) Amazon EBS & Cassandra: 1 Million Writes Per SecondAmazon Web Services
With the introduction of Amazon Elastic Block Store (EBS) GP2 and recent stability improvements, EBS has gained credibility in the Cassandra world for high performance workloads. By running Cassandra on Amazon EBS, you can run denser, cheaper Cassandra clusters with just as much availability as ephemeral storage instances. This talk walks through a highly detailed use case and configuration guide for a multi PetaByte, million write per second cluster that needs to be high performing and cost efficient. We explore the instance type choices, configuration, and low-level tuning that allowed us to hit 1.3 million writes per second with a replication factor of 3 on just 60 nodes.
Golang é uma linguagem fantástica para se desenvolver aplicações e um fator a ser explorado é o seu uso em dispositivos IoT. A linguagem já conta com diversas ferramentas de cross-compile, alguns pacotes experimentais de comunicação baixo nível e diversos projetos relacionados a hardware.
While probably the most prominent, Docker is not the only tool for building and managing containers. Originally meant to be a "chroot on steroids" to help debug systemd, systemd-nspawn provides a fairly uncomplicated approach to work with containers. Being part of systemd, it is available on most recent distributions out-of-the-box and requires no additional dependencies.
This deck will introduce a few concepts involved in containers and will guide you through the steps of building a container from scratch. The payload will be a simple service, which will be automatically activated by systemd when the first request arrives.
1 Million Writes per second on 60 nodes with Cassandra and EBSJim Plush
EBS has long been taboo in the Cassandra world for high performance workloads. That line of thinking has started to change with the introduction of EBS GP2 and the recent stability improvements made by the EBS team, which is why we have multiple PetaBytes of data relying on EBS every day. Running Cassandra on EBS will now let you run denser, cheaper Cassandra clusters with just as much availability as ephemeral storage instances. This talk will walk through a highly detailed use case and configuration guide for a multi PetaByte, million write per second cluster that needs to be highly performant and cost efficient. We will dive into the instance type choices, configuration and low level tuning that allowed us to hit 1.3 million writes per second with a replication factor of 3 on just 60 nodes. We will go into the details of why we chose to use the latest DateTieredCompactionStrategy and why that's a perfect fit for high volume time series workloads.
A gateway server is a server through which the computers in a LAN access the Internet. This is
usually done through NAT. It should also provide firewall protection for the LAN and it can also serve
as a DNS and DHCPD server for the LAN. Some years ago I have been involved in a project for building gateway servers like this, using
slackware on old PCs. In this article I will try to explain the things that I have done on this project and
how I did them.
The detail architecture of the most relevant consumer drones will be introduced, continuing with the communications protocol between the pilot (app in the smartphone or remote controller) and the drone. Manual reverse engineering on the binary protocol used for this communication will lead to identifying and understanding all the commands from each of the drones, and later inject commands back.
Learning Objectives:
1: Understand whenever a protocol between drone and pilot is secure.
2: Learn about a new reverse engineering methodology for these protocols.
3: Review a set of good practices to secure the environment surrounding a drone.
(Source: RSA Conference USA 2018)
Remix of two other open source presentations along with my own content, 40 slides set to play at 20 seconds auto-timed (similar to Pecha-Kucha style timing). This was delivered via Caribbean Tech Dev forum's monthly Google Hangout in November 2015, and video can be viewed at https://www.youtube.com/watch?v=xANrsSin_-0
Swift Install Workshop - OpenStack Conference Spring 2012Joe Arnold
OpenStack Swift is a highly-available distributed object storage
system which supports highly concurrent workloads. Swift is the
backbone behind Cloud Files, Rackspace's storage-as-a-service
offering.
In this workshop, which will be hosted by members of SwiftStack, Inc.,
we'll walk you through deployment and use of OpenStack Swift. We'll
begin by showing you how to install Swift from the ground up.
You'll learn:
- what you should know about Swift's architecture
- how to bootstrap a basic Swift installation
After that, we'll cover how to use Swift, including information on:
- creating accounts and users
- adding, removing, and managing data
- building applications on top of Swift
Bring your laptop (with virutalization extensions enabled in the BIOS)
and we will walk through setting up Swift in a virtual machine. We'll
also build an entire application on top of Swift to illustrate how to
use Swift as a storage service. This is a workshop you won't want to
miss!
This talk will focus on a brief overview of Kubernetes, with a brief demo, and then more of an in-depth focus on issues we've faced moving PHP projects into Docker and Kubernetes like signal propagation, init systems, and logging.
Talk from Cape Town PHP meetup on Feb. 7, 2016:
https://www.meetup.com/Cape-Town-PHP-Group/events/237226310/
Code: https://github.com/zoidbergwill/kubernetes-php-examples
Slides as markdown: http://www.zoidbergwill.com/presentations/2017/kubernetes-php/index.md
Asiste a este evento en línea para aprender un poco sobre Runtime Fabric. Juan Ore, arquitecto en Bits In Glass, nos hablará sobre los aspectos básicos de este servicio de MuleSoft, y hará un pequeño demo para verlo en acción. También veremos algunas transformaciones más complejas de DataWeave 2.0 para expandir nuestro conocimiento.
Similar to Trying and evaluating the new features of GlusterFS 3.5 (20)
One of the presentations used in a discussion meeting about GlusterFS held on Sep. 14, 2011 in Japan.
Ust: http://www.ustream.tv/channel/glusterfs
Togetter: http://togetter.com/li/188183
One of the presentations used in a discussion meeting about GlusterFS held on Sep. 14, 2011 in Japan.
Ust: http://www.ustream.tv/channel/glusterfs
Togetter: http://togetter.com/li/188183
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
2. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 2
Agenda
! About me
! The new features
! Additional news
! Conclusion
3. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 3
About
Me
4. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 4
About me
! Work at as a ...
! Programmer
! (Most recently)
! Also Interested in
! Software Engineer
! Cloud Computing
! Big Data/Data Science
! Something new technologies
! Supporting GlusterFS/Red Hat Storage Introduction
with Red Hat K.K.
5. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 5
About me
! GlusterFS since 2007 (v1.3.7)
! for my internet crawler at first.
! Love Gluster because of the ...
! Potential
! Performance
! Code
! Community
! Introduced or introducing it into ...
! Printer and scanner solution (field trial)
! Email services
! File storage services (WebDAV, NFS)
! Backup services
! Shared storage platform
! Medical service
! A board member of the Gluster
Community
6. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 6
My system
Seg.1: 192.168.79.0/24, GigE
Seg.2: 10.0.0.0/8, 100BaseT(USB Ethernet)
eins
zwei
drei
vier
fuenf
sechs
sieben
.1
.2
.3
.4
.5
.6
.7
.79.0.1
.79.0.2
.79.0.3
.79.0.4
.79.0.5
.79.0.6
• Seven nodes, connected to two separated physical network
segments.
• Seg.1 is for GlusterFS and Seg.2 is for other purposes (e.g. SSH)
• Each node is setup with:
• CentOS 6.5 x86_64
• GlusterFS 3.5.0 (from source tarball)
.79.0.7
storage
pool
(mainly)
client
7. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 7
My system
• Intel NUC DN2820FYKH
• Celeron2.4GHz dual-core, 1MB cache
• 8GB RAM
• 1TB Solid-state hard drive (w/ 8GB SLC SSD)
• 7.5W TDP
• Why?
• Separate several
loads (mainly of disk
accesses and
network traffics)
• Enough cheap to
build (38k JPY/node)
• Save money on
electricity (2 JPY/d/
node)
• Suppress my room's
temperature
increasing
8. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 8
My system
% sudo yum install -y openssh-clients make rpm-build bison flex automake libtool ncurses-
devel readline-devel openssl-devel libxml2-devel libibverbs-devel libacl-devel libattr-devel
python-devel python-setuptools lvm2-devel systemtap-sdt-devel libaio-devel xfsprogs glib2-
devel
% tar xzf glusterfs-3.5.0.tar.gz && cd glusterfs-3.5.0
% ./configure --prefix=/usr/local/glusterfs-3.5.0 --enable-bd-xlator --enable-fusermount --
enable-systemtap --enable-debug --enable-crypt-xlator --enable-qemu-block --enable-glupy
% make && sudo make install
# ln -sfn /usr/local/glusterfs-3.5.0 /usr/local/glusterfs
# cp -p /etc/init.d/glusterd /etc/init.d/glusterd-3.5.0
# cat <<EOF >> ~/.zshrc
export PATH=$PATH:/usr/local/glusterfs/sbin
export MANPATH=$MANPATH:/usr/local/glusterfs/share/man
EOF
# source ~/.zshrc
# echo "/usr/local/glusterfs/lib" > /etc/ld.so.conf.d/glusterfs.conf
# ldconfig
# sed -i 's/SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
# chkconfig iptables off
# /etc/init.d/iptables stop
GlusterFS 3.5.0 was installed on each node
in following way:
9. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 9
12 new features
10. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 10
Overview
Features Open
Stack
Opera-
tion
Mana-
gement
Scala-
bility
Perfor-
mance
Stab-
ility
Sec-
urity
Dev
AFR_CLI_enhancements ✔️
Exposing Volume
Capabilities
✔️
File Snapshot ✔️
GFID Access ✔️
On-Wire Compression +
Decompression
✔️
Prevent NFS restart on
Volume change (Part 1)
✔️
Quota Scalability ✔️ ✔️
readdir_ahead ✔️
zerofill ✔️ ✔
Brick Failure Detection ✔️
Disk encryption ✔️
Geo-Replication
Enhancement
✔ ✔️
11. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 11
OpenStack
Integration
Enhancements
12. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 12
File Snapshot
# setfattr -n
trusted.glusterfs.block-format
-v qcow2:<file_size(in KB/MB/GB)>
<file_name>
features/
qemu-block
xlator
# setfattr -n
trusted.glusterfs.block-snapshot-create
-v <snapshot_name1> <file_name>
# setfattr -n
trusted.glusterfs.block-snapshot-create
-v <snapshot_name2> <file_name>
# setfattr -n
trusted.glusterfs.block-snapshot-goto
-v <snapshot_name1> <file_name>
# setfattr -n
trusted.glusterfs.block-snapshot-delete
-v <snapshot_name2> <file_name>
a file
<file_name>
under
a mount
point
of a volume
fuse
hook
to
glusterfs
client
process
Restore from a snapshot
Take a snapshot
Take a snapshot
13. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 13
File Snapshot
features/
qemu-block
xlator
a file
<file_name>
under
a mount
point
of a volume
as a block
storage
for
Cinder
fuse
hook
to
glusterfs
client
process
Restore from a snapshot
Take a snapshot
Take a snapshot
OpenStack
Cinder
BD
xlator
block-format
block-snapshot-create
block-snapshot-create
block-snapshot-goto
block-snapshot-delete
14. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 14
zerofill
glusterfsdglusterfsd
AFR
libgfapi
User App
(e.g. Cinder)
0000
0000
0000
0000
0000
0000
posix_do_zerofill function
ZEROFILL fop
(glfs_zerofill function)
SCSI WRITESAME
command
BLKZEROOUT ioctl on Linux
15. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 15
zerofill
Server offloaded zerofill vs repeated zeroing
[root@llmvm02 remote]# time ./offloaded aakash-test log 20
real 3m34.155s
user 0m0.018s
sys 0m0.040s
[root@llmvm02 remote]# time ./manually aakash-test log 20
real 4m23.043s
user 0m2.197s
sys 0m14.457s
[root@llmvm02 remote]# time ./offloaded aakash-test log 25;
real 4m28.363s
user 0m0.021s
sys 0m0.025s
[root@llmvm02 remote]# time ./manually aakash-test log 25
real 5m34.278s
user 0m2.957s
sys 0m18.808s
http://www.gluster.org/community/documentation/index.php/Features/zerofill
1.23 times faster!
1.25 times faster!
16. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 16
Operation
Enhancements
17. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 17
AFR_CLI_enhancements
Before 3.5.0
# gluster volume heal vol1
Heal operation on volume vol1 has been successful
# gluster volume heal vol1 info
...
# gluster volume heal vol1 info healed
...
# gluster volume heal vol1 info heal-failed
...
# gluster volume heal vol1 info split-brain
...
Too many operations to know all
the situations...
What I want to know is not the
file names...
How long the healing takes?
I don't know when the split-brain
detected but...
18. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 18
AFR_CLI_enhancements
After 3.5.0
# gluster volume heal vol1 statistics
Gathering crawl statistics on volume vol1 has been successful
------------------------------------------------
Crawl statistics for brick no 0
Hostname of brick eins
Starting time of crawl: Mon May 19 10:13:02 2014
Ending time of crawl: Mon May 19 10:13:02 2014
Type of crawl: INDEX
No. of entries healed: 0
No. of entries in split-brain: 0
No. of heal failed entries: 0
...
Wow! I can get the statistic and historical
information at a glance!
19. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 19
Management
Enhancements
20. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 20
# gluster volume info
Volume Name: bd0
Type: Distribute
Volume ID: 019d0f4b-d11a-480e-9be8-0c79902f0746
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: sieben:/tmp/bd0-meta
Exposing Volume Capabilities
I confuse which volume type the volume supports.
So I should manage it with other tools like Excel...
Before 3.5.0
21. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 21
# gluster volume info
Volume Name: bd0
Type: Distribute
Volume ID: 019d0f4b-d11a-480e-9be8-0c79902f0746
Status: Started
Xlator 1: BD
Capability 1: thin
Capability 2: offload_copy
Capability 3: offload_snapshot
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: sieben:/tmp/bd0-meta
Brick1 VG: bd0-vg
Exposing Volume Capabilities
Probe the type of
volume
Provide list of
capabilities of a
xlator/volume.
Yeah! I can understand the volume type
and the detail!
After 3.5.0
22. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 22
Review: How to use BD xlator
# dd if=/dev/zero of=/tmp/bd-loop6 bs=1M count=2048
# losetup /dev/loop6 /tmp/bd-loop6
# pvcreate /dev/loop6
# vgcreate bd0-vg /dev/loop6
Volume group "bd0-vg" successfully created
# lvcreate --thin bd0-vg -L 1000M
Logical volume "lvol0" created
Logical volume "lvol1" created
This VG becomes a
volume of GlusterFS
If you want to get the BDs
thin-provisioned ones, hit
the lvcreate command.
(And the names are fixed.)
Here created a VG with
a single 2GB of PV
23. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 23
Review: How to use BD xlator
# lvdisplay bd0-vg
--- Logical volume ---
LV Name lvol1
VG Name bd0-vg
LV UUID PSAFkr-Vyr8-fkGU-kDnA-rWUF-fFFT-111Snr
LV Write Access read/write
LV Creation host, time sieben, 2014-05-18 14:38:21 +0900
LV Pool transaction ID 0
LV Pool metadata lvol1_tmeta
LV Pool data lvol1_tdata
LV Pool chunk size 64.00 KiB
LV Zero new blocks yes
LV Status available
# open 0
LV Size 1000.00 MiB
Allocated pool data 0.00%
Allocated metadata 0.88%
Current LE 250
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:5
A logical volume pool for
thin-provisioning.
No need when using no
thin-provisioning.
24. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 24
Review: How to use BD xlator
# mkdir /tmp/bd0-meta
# gluster volume create bd0 sieben:/tmp/bd0-meta?bd0-vg force
volume create: bd0: success: please start the volume to access data
# gluster volume start bd0
volume start: bd0: success
# gluster volume info bd0
Volume Name: bd0
Type: Distribute
Volume ID: 019d0f4b-d11a-480e-9be8-0c79902f0746
Status: Started
Xlator 1: BD
Capability 1: thin
Capability 2: offload_copy
Capability 3: offload_snapshot
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: sieben:/tmp/bd0-meta
Brick1 VG: bd0-vg
# mkdir /mnt/glusterfs/bd0
# mount-t glusterfs sieben:/bd0 /mnt/glusterfs/bd0
Meta data store
for BD xlator
"?" (question
mark) is the
separator
25. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 25
Review: How to use BD xlator
# touch /mnt/glusterfs/bd0/lv0
# setfattr -n "user.glusterfs.bd" -v "thin:1024MB" /mnt/glusterfs/bd0/lv0
# lvdisplay bd0-vg
--- Logical volume ---
LV Name lvol1
VG Name bd0-vg
LV UUID PSAFkr-Vyr8-fkGU-kDnA-rWUF-fFFT-111Snr
LV Write Access read/write
LV Creation host, time sieben.infinibridge.net, 2014-05-18 14:38:21 +0900
LV Pool transaction ID 1
LV Pool metadata lvol1_tmeta
LV Pool data lvol1_tdata
LV Pool chunk size 64.00 KiB
LV Zero new blocks yes
LV Status available
# open 0
LV Size 1000.00 MiB
Allocated pool data 0.00%
Allocated metadata 0.98%
Current LE 250
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:5
Create a file that
is backed by an
LV
Or simply -v "lv" when
no need for thin-
provisioning
26. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 26
Review: How to use BD xlator
--- Logical volume ---
LV Path /dev/bd0-vg/a9790eba-ffbf-4d9c-a674-e02c61ece935
LV Name a9790eba-ffbf-4d9c-a674-e02c61ece935
VG Name bd0-vg
LV UUID Z4HtWM-W0jk-YiK5-66ED-zOMw-YhFp-nrnRUU
LV Write Access read/write
LV Creation host, time sieben.infinibridge.net, 2014-05-18 14:47:31 +0900
LV Pool name lvol1
LV Status available
# open 0
LV Size 1.00 GiB
Mapped size 0.00%
Current LE 256
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:9
27. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 27
Review: How to use BD xlator
# for i in `seq 1 9`; do touch /mnt/glusterfs/bd0/lv$i; setfattr -n "user.glusterfs.bd" -v
"thin:1024MB" /mnt/glusterfs/bd0/lv$i; done
# lvdisplay -C bd0-vg
LV VG Attr LSize Pool Origin Data% Move
Log Cpy%Sync Convert
39b82644-f8ef-435d-b14e-d199a7e264fa bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
6002ddb2-28f1-463c-8666-f683fe2441ed bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
69993340-d691-4502-a9d5-375b8be0fb9e bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
82af50a2-0124-41d8-a887-d8c30427a663 bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
996969dd-3e32-491b-95d1-f279e6808d5b bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
a19ac2af-94df-4d01-b7c3-bbfcbfe5d09e bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
a9790eba-ffbf-4d9c-a674-e02c61ece935 bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
d6fd964a-67f8-4d48-96d1-343bed4ee792 bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
ea58b011-3a41-4bf0-9fe6-3862e24b86f6 bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
f7df48e5-09b1-4314-b729-1f38e5ceec2e bd0-vg Vwi-a-tz-- 1.00g lvol1 0.00
lvol1 bd0-vg twi-a-tz-- 1000.00m 0.00
Here we create other
nine LVs in the same
way.
28. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 28
Review: How to use BD xlator
# mkdir /mnt/bd0-lv/{39b82644-f8ef-435d-b14e-d199a7e264fa,6002ddb2-28f1-463c-8666-
f683fe2441ed,69993340-d691-4502-a9d5-375b8be0fb9e,82af50a2-0124-41d8-a887-
d8c30427a663,996969dd-3e32-491b-95d1-f279e6808d5b,a19ac2af-94df-4d01-b7c3-
bbfcbfe5d09e,a9790eba-ffbf-4d9c-a674-
e02c61ece935,d6fd964a-67f8-4d48-96d1-343bed4ee792,ea58b011-3a41-4bf0-9fe6-3862e24b86f6,f7df4
8e5-09b1-4314-b729-1f38e5ceec2e}
# ls /mnt/bd0-lv
39b82644-f8ef-435d-b14e-d199a7e264fa a19ac2af-94df-4d01-b7c3-bbfcbfe5d09e
6002ddb2-28f1-463c-8666-f683fe2441ed a9790eba-ffbf-4d9c-a674-e02c61ece935
69993340-d691-4502-a9d5-375b8be0fb9e d6fd964a-67f8-4d48-96d1-343bed4ee792
82af50a2-0124-41d8-a887-d8c30427a663 ea58b011-3a41-4bf0-9fe6-3862e24b86f6
996969dd-3e32-491b-95d1-f279e6808d5b f7df48e5-09b1-4314-b729-1f38e5ceec2e
# for x in 39b82644-f8ef-435d-b14e-d199a7e264fa 6002ddb2-28f1-463c-8666-f683fe2441ed
69993340-d691-4502-a9d5-375b8be0fb9e 82af50a2-0124-41d8-a887-d8c30427a663
996969dd-3e32-491b-95d1-f279e6808d5b a19ac2af-94df-4d01-b7c3-bbfcbfe5d09e a9790eba-
ffbf-4d9c-a674-e02c61ece935 d6fd964a-67f8-4d48-96d1-343bed4ee792
ea58b011-3a41-4bf0-9fe6-3862e24b86f6 f7df48e5-09b1-4314-b729-1f38e5ceec2e; do mkfs.xfs -i
size=512 /dev/bd0-vg/$x && mount -t xfs /dev/bd0-vg/$x /mnt/bd0-lv/$x; done
Creating mount
point for each LV.
Formatting each LV
in XFS and mount it.
29. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 29
Review: How to use BD xlator
# df -h | grep bd0-lv
/dev/dm-13 1014M 33M 982M 4% /mnt/bd0-lv/39b82644-f8ef-435d-b14e-d199a7e264fa
/dev/dm-16 1014M 33M 982M 4% /mnt/bd0-lv/6002ddb2-28f1-463c-8666-f683fe2441ed
/dev/dm-18 1014M 33M 982M 4% /mnt/bd0-lv/69993340-d691-4502-a9d5-375b8be0fb9e
/dev/dm-11 1014M 33M 982M 4% /mnt/bd0-lv/82af50a2-0124-41d8-a887-d8c30427a663
/dev/dm-12 1014M 33M 982M 4% /mnt/bd0-lv/996969dd-3e32-491b-95d1-f279e6808d5b
/dev/dm-17 1014M 33M 982M 4% /mnt/bd0-lv/a19ac2af-94df-4d01-b7c3-bbfcbfe5d09e
/dev/dm-9 1014M 33M 982M 4% /mnt/bd0-lv/a9790eba-ffbf-4d9c-a674-e02c61ece935
/dev/dm-14 1014M 33M 982M 4% /mnt/bd0-lv/d6fd964a-67f8-4d48-96d1-343bed4ee792
/dev/dm-15 1014M 33M 982M 4% /mnt/bd0-lv/ea58b011-3a41-4bf0-9fe6-3862e24b86f6
/dev/dm-10 1014M 33M 982M 4% /mnt/bd0-lv/f7df48e5-09b1-4314-b729-1f38e5ceec2e
# mount | grep bd0-lv
/dev/mapper/bd0--vg-39b82644--f8ef--435d--b14e--d199a7e264fa on /mnt/bd0-lv/39b82644-f8ef-435d-b14e-d199a7e264fa type xfs (rw)
/dev/mapper/bd0--vg-6002ddb2--28f1--463c--8666--f683fe2441ed on /mnt/bd0-lv/6002ddb2-28f1-463c-8666-f683fe2441ed type xfs (rw)
/dev/mapper/bd0--vg-69993340--d691--4502--a9d5--375b8be0fb9e on /mnt/bd0-lv/69993340-d691-4502-a9d5-375b8be0fb9e type xfs (rw)
/dev/mapper/bd0--vg-82af50a2--0124--41d8--a887--d8c30427a663 on /mnt/bd0-lv/82af50a2-0124-41d8-a887-d8c30427a663 type xfs (rw)
/dev/mapper/bd0--vg-996969dd--3e32--491b--95d1--f279e6808d5b on /mnt/bd0-lv/996969dd-3e32-491b-95d1-f279e6808d5b type xfs (rw)
/dev/mapper/bd0--vg-a19ac2af--94df--4d01--b7c3--bbfcbfe5d09e on /mnt/bd0-lv/a19ac2af-94df-4d01-b7c3-bbfcbfe5d09e type xfs (rw)
/dev/mapper/bd0--vg-a9790eba--ffbf--4d9c--a674--e02c61ece935 on /mnt/bd0-lv/a9790eba-ffbf-4d9c-a674-e02c61ece935 type xfs (rw)
/dev/mapper/bd0--vg-d6fd964a--67f8--4d48--96d1--343bed4ee792 on /mnt/bd0-lv/d6fd964a-67f8-4d48-96d1-343bed4ee792 type xfs (rw)
/dev/mapper/bd0--vg-ea58b011--3a41--4bf0--9fe6--3862e24b86f6 on /mnt/bd0-lv/ea58b011-3a41-4bf0-9fe6-3862e24b86f6 type xfs (rw)
/dev/mapper/bd0--vg-f7df48e5--09b1--4314--b729--1f38e5ceec2e on /mnt/bd0-lv/f7df48e5-09b1-4314-b729-1f38e5ceec2e type xfs (rw)
'Cause of thin-provisioning, in
total 10GB of block devices are
created on the 2GB of VG!
30. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 30
Review: How to use BD xlator
[sechs]# mount -t glusterfs localhost:/bd0 /mnt/glusterfs/bd0
[sechs]# mount -t xfs -o loop /mnt/glusterfs/bd0/lv0
[sechs]# df -h | grep bd0-lv
1014M 33M 982M 4% /mnt/bd0-lv/lv1
The block devices are shared
with GlusterFS as files.
raw block device
physical volume
volume group
LV LV LV
BD volume=
file file file
Convert them with lvm2
development library
=
Shared with
GlusterFS
Snapshot and
clone are
capable as LV
31. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 31
Brick Failure Detection
Before 3.5.0
1. One of the
backend storage
failed!
2. R/W ops from
a client
glusterfsdglusterfsd
AFR
3. glusterfsd returned
"Input/output error" or
"Read-only filesystem"
directly.
32. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 32
Brick Failure Detection
After 3.5.0
1. One of the
backend storage
failed!
3. R/W ops from
a client
glusterfsdglusterfsd
AFR
4. The client gets no
error and completes
the operation.
2. glusterfsd outputs logs
and shutdowns itself.
33. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 33
Brick Failure Detection
# brick="/mnt/lv4/vol4"; gluster volume create vol4 eins:$brick zwei:$brick drei:$brick
vier:$brick fuenf:$brick sechs:$brick
# gluster volume start vol4
# gluster volume set vol4 storage.health-check-interval 10
# gluster volume info vol4
Volume Name: vol4
Type: Distribute
Volume ID: 706122a9-44fc-4d1d-8c3b-97482d98b95c
Status: Started
Number of Bricks: 6
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv4/vol4
Brick2: zwei:/mnt/lv4/vol4
Brick3: drei:/mnt/lv4/vol4
Brick4: vier:/mnt/lv4/vol4
Brick5: fuenf:/mnt/lv4/vol4
Brick6: sechs:/mnt/lv4/vol4
Options Reconfigured:
storage.health-check-interval: 10
Setup for test
34. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 34
Brick Failure Detection
[sechs]# dmsetup table
vg0-swift: 0 209715200 linear 8:7 838862848
vg0-cinder: 0 209715200 linear 8:7 419432448
vg0-lv4: 0 209715200 linear 8:7 1468008448
vg0-lv3: 0 209715200 linear 8:7 1258293248
vg0-lv2: 0 209715200 linear 8:7 1048578048
vg0-lv1: 0 209715200 linear 8:7 209717248
vg0-lv0: 0 209715200 linear 8:7 2048
vg0-glance: 0 209715200 linear 8:7 629147648
Setup for test (contd.)
35. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 35
Brick Failure Detection
[sechs]# echo 0 209715200 error > dmsetup-error-target
[sechs]# dmsetup load vg0-lv4 dmsetup-error-target
[sechs]# dmsetup resume vg0-lv4
[sechs]# dmsetup table
vg0-swift: 0 209715200 linear 8:7 838862848
vg0-cinder: 0 209715200 linear 8:7 419432448
vg0-lv4: 0 209715200 error
vg0-lv3: 0 209715200 linear 8:7 1258293248
vg0-lv2: 0 209715200 linear 8:7 1048578048
vg0-lv1: 0 209715200 linear 8:7 209717248
vg0-lv0: 0 209715200 linear 8:7 2048
vg0-glance: 0 209715200 linear 8:7 629147648
Brick failure test
36. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 36
Brick Failure Detection
[2014-05-18 18:49:53.720594] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file
changed
[2014-05-18 18:50:04.238239] W [posix-helpers.c:1294:posix_health_check_thread_proc] 0-vol4-
posix: stat() on /mnt/lv4/vol4 returned: Input/output error
[2014-05-18 18:50:04.238328] M [posix-helpers.c:1314:posix_health_check_thread_proc] 0-vol4-
posix: health-check failed, going down
Message from syslogd@sechs at May 19 03:50:04 ...
glusterfsd: [2014-05-18 18:50:04.238328] M [posix-helpers.c:
1314:posix_health_check_thread_proc] 0-vol4-posix: health-check failed, going down
[2014-05-18 18:50:34.238551] M [posix-helpers.c:1319:posix_health_check_thread_proc] 0-vol4-
posix: still alive! -> SIGTERM
Message from syslogd@sechs at May 19 03:50:34 ...
glusterfsd: [2014-05-18 18:50:34.238551] M [posix-helpers.c:
1319:posix_health_check_thread_proc] 0-vol4-posix: still alive! -> SIGTERM
[2014-05-18 18:50:34.238910] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib64/libc.so.
6(clone+0x6d) [0x7f1144ebab7d] (-->/lib64/libpthread.so.0(+0x79d1) [0x7f114554d9d1] (-->/
usr/local/glusterfs-3.5.0/sbin/glusterfsd(glusterfs_sigwaiter+0xf0) [0x4085af]))) 0-:
received signum (15), shutting down
var/log/glusterfs/bricks/mnt-lv4-vol4.log on the failed node
37. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 37
Brick Failure Detection
May 19 03:49:55 sechs kernel: XFS (dm-7): metadata I/O error: block 0x0
("xfs_buf_iodone_callbacks") error 5 buf count 4096
May 19 03:49:57 sechs kernel: XFS (dm-7): metadata I/O error: block 0x6400108
("xlog_iodone") error 5 buf count 4096
May 19 03:49:57 sechs kernel: XFS (dm-7): xfs_do_force_shutdown(0x2) called from line 1062
of file fs/xfs/xfs_log.c. Return address = 0xffffffffa04dd131
May 19 03:49:57 sechs kernel: XFS (dm-7): Log I/O Error Detected. Shutting down filesystem
May 19 03:49:57 sechs kernel: XFS (dm-7): Please umount the filesystem and rectify the
problem(s)
May 19 03:50:04 sechs glusterfsd: [2014-05-18 18:50:04.238328] M [posix-helpers.c:
1314:posix_health_check_thread_proc] 0-vol4-posix: health-check failed, going down
Message from syslogd@sechs at May 19 03:50:04 ...
glusterfsd: [2014-05-18 18:50:04.238328] M [posix-helpers.c:
1314:posix_health_check_thread_proc] 0-vol4-posix: health-check failed, going down
May 19 03:50:27 sechs kernel: XFS (dm-7): xfs_log_force: error 5 returned.
Message from syslogd@sechs at May 19 03:50:34 ...
glusterfsd: [2014-05-18 18:50:34.238551] M [posix-helpers.c:
1319:posix_health_check_thread_proc] 0-vol4-posix: still alive! -> SIGTERM
May 19 03:50:34 sechs glusterfsd: [2014-05-18 18:50:34.238551] M [posix-helpers.c:
1319:posix_health_check_thread_proc] 0-vol4-posix: still alive! -> SIGTERM
May 19 03:50:57 sechs kernel: XFS (dm-7): xfs_log_force: error 5 returned.
syslog on the failed node
38. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 38
Brick Failure Detection
# gluster volume status vol4
Status of volume: vol4
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick eins:/mnt/lv4/vol4 49160 Y 2925
Brick zwei:/mnt/lv4/vol4 49159 Y 440
Brick drei:/mnt/lv4/vol4 49152 Y 32500
Brick vier:/mnt/lv4/vol4 49152 Y 32657
Brick fuenf:/mnt/lv4/vol4 49152 Y 24517
Brick sechs:/mnt/lv4/vol4 N/A N N/A
NFS Server on localhost 2049 Y 29535
NFS Server on zwei N/A N N/A
NFS Server on vier N/A N N/A
NFS Server on drei N/A N N/A
NFS Server on eins N/A N N/A
NFS Server on fuenf N/A N N/A
NFS Server on sechs N/A N N/A
Task Status of Volume vol4
------------------------------------------------------------------------------
There are no active volume tasks
gluster volume status
39. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 39
Brick Failure Detection
# ps -ef | grep glusterfsd | grep -v grep | wc -l
0
processes on the failed node
40. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 40
Brick Failure Detection
[sechs]# service glusterd restart
restart glusterd (and glusterfsd) on the failed node
[2014-05-18 18:58:17.197872] I [glusterfsd.c:1959:main] 0-/usr/local/glusterfs-3.5.0/sbin/
glusterfsd: Started running /usr/local/glusterfs-3.5.0/sbin/glusterfsd version 3.5git (/usr/
local/glusterfs-3.5.0/sbin/glusterfsd -s sechs --volfile-id vol4.sechs.mnt-lv4-vol4 -p /var/
lib/glusterd/vols/vol4/run/sechs-mnt-lv4-vol4.pid -S /var/run/
23afc72b5ceddccd28b405b1cdf5b4df.socket --brick-name /mnt/lv4/vol4 -l /usr/local/
glusterfs-3.5.0/var/log/glusterfs/bricks/mnt-lv4-vol4.log --xlator-option *-posix.glusterd-
uuid=0765d288-a59b-4ccf-90ae-c3332c83dbf4 --brick-port 49152 --xlator-option vol4-
server.listen-port=49152)
[2014-05-18 18:58:17.205310] I [socket.c:3561:socket_init] 0-socket.glusterfsd: SSL support
is NOT enabled
[2014-05-18 18:58:17.205486] I [socket.c:3576:socket_init] 0-socket.glusterfsd: using system
polling thread
[2014-05-18 18:58:17.205880] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT
enabled
[2014-05-18 18:58:17.205949] I [socket.c:3576:socket_init] 0-glusterfs: using system polling
thread
[2014-05-18 18:58:18.834910] I [graph.c:254:gf_add_cmdline_options] 0-vol4-server: adding
option 'listen-port' for volume 'vol4-server' with value '49152'
41. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 41
Brick Failure Detection
[2014-05-18 18:58:18.834976] I [graph.c:254:gf_add_cmdline_options] 0-vol4-posix: adding
option 'glusterd-uuid' for volume 'vol4-posix' with value '0765d288-a59b-4ccf-90ae-
c3332c83dbf4'
[2014-05-18 18:58:18.837332] I [rpcsvc.c:2064:rpcsvc_set_outstanding_rpc_limit] 0-rpc-
service: Configured rpc.outstanding-rpc-limit with value 64
[2014-05-18 18:58:18.837510] W [options.c:848:xl_opt_validate] 0-vol4-server: option
'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with
correction
[2014-05-18 18:58:18.837572] I [socket.c:3561:socket_init] 0-tcp.vol4-server: SSL support is
NOT enabled
[2014-05-18 18:58:18.837601] I [socket.c:3576:socket_init] 0-tcp.vol4-server: using system
polling thread
[2014-05-18 18:58:18.838445] E [common-utils.c:93:mkdir_p] 0-: Failed due to reason Input/
output error
[2014-05-18 18:58:18.838505] I [mem-pool.c:539:mem_pool_destroy] 0-vol4-changelog: size=108
max=0 total=0
[2014-05-18 18:58:18.838533] E [xlator.c:403:xlator_init] 0-vol4-changelog: Initialization
of volume 'vol4-changelog' failed, review your volfile again
[2014-05-18 18:58:18.838561] E [graph.c:307:glusterfs_graph_init] 0-vol4-changelog:
initializing translator failed
[2014-05-18 18:58:18.838610] E [graph.c:502:glusterfs_graph_activate] 0-graph: init failed
restart glusterd (and glusterfsd) on the failed node (contd.)
42. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 42
Brick Failure Detection
[2014-05-18 18:58:18.839480] W [glusterfsd.c:1095:cleanup_and_exit] (-->/usr/local/
glusterfs-3.5.0/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0x1b5) [0x7f2981c837d8] (-->/usr/
local/glusterfs-3.5.0/sbin/glusterfsd(mgmt_getspec_cbk+0x36a) [0x40cf77] (-->/usr/local/
glusterfs-3.5.0/sbin/glusterfsd(glusterfs_process_volfp+0x18a) [0x408bf2]))) 0-: received
signum (0), shutting down
restart glusterd (and glusterfsd) on the failed node (contd.)
43. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 43
Scalability
Enhancement
44. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 44
Quota Scalability
Before 3.5.0
Directory Quota limitation
= a few hundreds per volume
45. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 45
Quota Scalability
After 3.5.0
Directory Quota limitation
= 65536 per volume
46. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 46
Performance
Enhancements
47. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 47
On-Wire Compression + Decompression
FUSE client Storage pool
FUSE client Storage pool
Write ops
Read ops
3. Transport
2. Compression
1. open
and
write
4. Decompression
and
write to disk
1. open
and
read
4. Decompression 3. Transport
2. read and
Compression
48. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 48
On-Wire Compression + Decompression
# gluster volume create vol-comp eins:/mnt/lv3/vol-comp
# gluster volume set vol-comp network.compression on
# gluster volume set vol-comp network.compression.compression-level 8
# gluster volume set vol-comp network.compression.min-size 50
# gluster volume set vol-comp performance.write-behind off
# gluster volume set vol-comp performance.strict-write-ordering on
# gluster volume set vol-comp performance.open-behind off
# gluster volume info vol-comp
Volume Name: vol-comp
Type: Distribute
Volume ID: 92b47734-2552-4168-b3c3-151093562e4f
Status: Created
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv3/vol-comp
Options Reconfigured:
network.compression.min-size: 50
network.compression.compression-level: 8
performance.open-behind: off
performance.write-behind: off
performance.strict-write-ordering: on
network.compression.mode: server
network.compression: on
Data is
compressed only
when its size
exceeds the
above value in
bytes.
-1: default
compression (= 8)
0: no compression
1: best speed
9: best
compression
Turn off the
performance
translators
to avoid
Input/output error
49. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 49
On-Wire Compression + Decompression
# gluster volume start vol-comp
# mount -t glusterfs localhost:/vol-comp /mnt/glusterfs/vol-comp
# dd if=/dev/zero of=/mnt/glusterfs/vol-comp/1gb.dat bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 33.8606 s, 31.7 MB/s
# diff /mnt/glusterfs/vol-comp/1gb.dat /tmp/1gb.dat
#
• CPU load on client becomes higher than the one without
network compression.
• Tcpdump showed the 1GB of zero compressed into non-zero
one.
• High-end CPU might show greater performance.
• There are still issues and limitations
• It cannot work with striped volumes.
• For glusterfs versions <= 3.5, it cannot work with AFR.
117 MB/s when no
compression
Compression and
Decompression
executed correctly
50. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 50
readdir_ahead
Before 3.5.0
volume
read-ahead
Sequential file access can be
fast, but sequential directory
access like "ls" cannot.
51. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 51
readdir_ahead
After 3.5.0
volume
Sequential reads of large
directories can complete
faster!
readdir-aheadread-ahead
52. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 52
readdir_ahead
# gluster volume set vol0 readdir-ahead enable
volume set: success
# gluster volume info vol0
Volume Name: vol0
Type: Distribute
Volume ID: cf9db2aa-5ee8-40c3-8ca9-8316ab31ba59
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv0/vol0
Brick2: zwei:/mnt/lv0/vol0
Options Reconfigured:
performance.readdir-ahead: enable
disabled by default
How-to
53. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 53
readdir_ahead
# brick="/mnt/lv4/vol4"; gluster volume create vol4 eins:$brick zwei:$brick drei:$brick
vier:$brick fuenf:$brick sechs:$brick
# gluster volume start vol4
# mount -t glusterfs localhost:/vol4 /mnt/glusterfs/vol4
# mkdir /mnt/glusterfs/vol4/manyfiles
# for a in `seq 0 9`; do for b in `seq 0 9`; do for c in `seq 0 9`; for d in `seq 0 9`; do
for e in `seq 0 9`; do for f in `seq 0 9`; do for g in `seq 0 9`; do for h in `seq 0 9`; do
for i in `seq 0 9`; do file="/mnt/glusterfs/vol4/manyfiles/8kb${a}${b}${c}${d}${e}${f}${g}${h}
${i}.dat"; echo ${file}; dd if=/dev/zero of=${file} bs=1K count=8; if [ $? -ne 0 ]; then
break; fi; done; done; done; done; done; done; done; done
...
^C
# df -ki /mnt/glusterfs/vol4
Filesystem Inodes IUsed IFree IUse% Mounted on
localhost:vol4 314572800 3394646 311178154 2% /mnt/glusterfs/vol4
# umount /mnt/glusterfs/vol4
Setup for evaluation
3 million 8K files
54. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 54
readdir_ahead
# mount -t glusterfs localhost:/vol4 /mnt/glusterfs/vol4
# for i in `seq 0 2`; do time ls /mnt/glusterfs/vol4/manyfiles > /dev/null; done
26.24s user 18.70s system 6% cpu 12:03.05 total
26.58s user 12.10s system 5% cpu 11:45.92 total
26.53s user 21.61s system 5% cpu 14:14.75 total
# umount /mnt/glusterfs/vol4
Evaluation
# gluster volume stop vol4 && gluster volume start vol4
# gluster volume set vol4 readdir-ahead enable
# mount -t glusterfs localhost:/vol4 /mnt/glusterfs/vol4
# for i in `seq 0 2`; do time ls /mnt/glusterfs/vol4/manyfiles > /dev/null; done
26.24s user 17.97s system 11% cpu 6:25.09 total
26.58s user 22.36s system 10% cpu 8:02.83 total
26.57s user 22.83s system 10% cpu 8:13.01 total
# gluster volume reset vol4
# umount /mnt/glusterfs/vol4 1.68 times faster!
55. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 55
Stability
Enhancements
56. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 56
Prevent NFS restart on Volume change (Part 1)
Gluster NFS Graph
nfs/server
option nfs3.vol4.volume-id 706122a9-44fc-4d1d-8c3b-97482d98b95c
option rpc-auth.addr.vol4.allow *
option nfs3.vol-gfid-access.volume-id 73abf812-4fff-42bd-822b-3036b72f060d
option rpc-auth.addr.vol-gfid-access.allow *
option nfs3.vol2.volume-id d0517697-5372-44a1-960f-6db0d988f3b2
option rpc-auth.addr.vol2.allow *
option nfs3.vol-comp.volume-id 92b47734-2552-4168-b3c3-151093562e4f
option rpc-auth.addr.vol-comp.allow *
option nfs3.vol1.volume-id ba03d1e6-a520-4e7f-ac4c-2440a205e80e
option rpc-auth.addr.vol1.allow *
option nfs3.vol0.volume-id cf9db2aa-5ee8-40c3-8ca9-8316ab31ba59
option rpc-auth.addr.vol0.allow *
option nfs.drc on
option nfs.nlm on
option nfs.dynamic-volumes on
vol0
debug/io-stats
vol0-write-behind
performance/write-behind
vol0-dht
cluster/distribute
vol0-client-0
protocol/client
vol0-client-1
protocol/client
vol1
debug/io-stats
vol1-write-behind
performance/write-behind
vol1-dht
cluster/distribute
vol1-client-0
protocol/client
vol1-client-1
protocol/client
vol2
debug/io-stats
vol2-write-behind
performance/write-behind
vol2-dht
cluster/distribute
vol2-client-0
protocol/client
vol2-client-1
protocol/client
Single nfs/server
exists on the top of
all the volumes
57. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 57
Vol2
My presentation last year (2013)
! NFS and Multi-tenancy
! 'nfs.rpc-auth-allow' for multi-tenancy
! some operations on a volume affect
IOs to other volumes
Vol1
Vol0
e.g.
gluster volume set ...
IO
IO
IO
Vol2
Vol1
Vol0
IO
IO
IO
58. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 58
Prevent NFS restart on Volume change (Part 1)
! "Some operations" on a volume
! gluster volume {set|reset} <volumeName> nfs.rpc-auth-allow
! gluster volume {start|stop} <volumeName>
! gluster volume add-brick
! gluster volume remove-brick <volumeName> <brick1> ... <brickn>
commit
60. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 60
Geo-Replication Enhancement
storage
pool
(a cluster)
gsyncd
Before 3.5.0
SPOF!
identify file
changes with
xattrs
directory crawl
with rsync
61. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 61
Geo-Replication Enhancement
storage
pool
(a cluster)
gsyncd
for each
peer
After 3.5.0
identify file changes with
changelog in memory
gsyncd
for each
peer
gsyncd
for each
peer
gsyncd
for each
peer
gsyncd
for each
peer
gsyncd
for each
peer
63. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 63
Geo-Replication Enhancement
# ls -a /mnt/lv0/vol0/.glusterfs/changelogs
. ..
Changelog (contd.)
No use without
gsyncd???
64. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 64
Security
Enhancement
65. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 65
Disk encryption
FUSE client Storage pool
FUSE client Storage pool
Write ops
Read ops
1. open
and
write
2. Encryption
3. Transport
4. Write the
encrypted data
to disk
1. open
and
read
4. Decryption 3. Transport
2. read from
underlying disks
66. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 66
Disk encryption
# gluster volume info
Volume Name: vol2
Type: Replicate
Volume ID: e0332771-a3c2-4fe5-980c-b3860cfe3baf
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv2/vol2
Brick2: zwei:/mnt/lv2/vol2
# gluster volume set vol2 encryption on
volume set: success
# for x in quick-read write-behind open-behind; do gluster volume set vol2 performance.$x
off; done
# gluster volume set vol2 encryption.master-key /var/lib/glusterd/vols/vol2/
encryption.master-key
# openssl rand -hex 32 > /var/lib/glusterd/vols/vol2/encryption.master-key
# gluster volume set vol2 encryption.data-key-size 512
Setup
67. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 67
Disk encryption
# gluster volume info
Volume Name: vol2
Type: Replicate
Volume ID: e0332771-a3c2-4fe5-980c-b3860cfe3baf
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv2/vol2
Brick2: zwei:/mnt/lv2/vol2
Options Reconfigured:
encryption.data-key-size: 512
encryption.master-key: /var/lib/glusterd/vols/vol2/encryption.master-key
performance.open-behind: off
performance.write-behind: off
performance.quick-read: off
features.encryption: on
# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/var/lib/glusterd/vols/vol2/
encryption.master-key localhost:/vol2 /mnt/glusterfs/vol2
Setup (contd.)
68. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 68
Disk encryption
# echo "test" > /mnt/glusterfs/vol2/test.txt
# cat /mnt/glusterfs/vol2/test.txt
test
[eins]# cat /mnt/lv2/vol2/test.txt
Zd??]K!q??tuv
[zwei]# cat /mnt/lv2/vol2/test.txt
Zd??]K!q??tuv
Encryption test
# dd if=/dev/zero of=/mnt/glusterfs/vol1/test.dat bs=1 count=32
# dd if=/dev/zero of=/mnt/glusterfs/vol2/test.dat bs=1 count=32
[eins]# dd if=/dev/zero of=/tmp/test.dat bs=1 count=32
[eins]# diff /tmp/test.dat /mnt/lv2/vol2/test.dat
Binary files /tmp/test.dat and /mnt/lv2/vol2/test.dat differ
[eins]# diff /tmp/test.dat /mnt/lv1/vol1/test.dat
#
# tcpdump -i eth0 -XX
Can see the
transported
zeroed data
fully encrypted.
ASCII files on the
bricks are
encrypted.
Binary files on the
bricks are also
encrypted.
Confirm that no use
of encryption never
encrypt the data, so
you can access the
raw data on several
bricks without
encryption.
69. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 69
Disk encryption
# dd if=/dev/zero of=/tmp/1gb.dat bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 3.61505 s, 297 MB/s
# diff3 /tmp/1gb.dat /mnt/glusterfs/vol1/1gb.dat /mnt/glusterfs/vol2/1gb.dat
#
Decryption test
Perfect!
70. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 70
Disk encryption
# dd if=/dev/zero of=/mnt/glusterfs/vol1/1gb.dat bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 18.4542 s, 58.2 MB/s
# dd if=/dev/zero of=/mnt/glusterfs/vol2/1gb.dat bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 263.633 s, 4.1 MB/s
Performance test
71. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 71
Disk encryption
# mount -t nfs -o vers=3,hard,intr,nosuid localhost:/vol2 /mnt/nfs/vol2
mount.nfs: Connection timed out
Work with NFS? (No!)
72. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 72
Disk encryption
# cp /var/lib/glusterd/vols/vol2/encryption.master-key /tmp
# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/tmp/encryption.master-key
localhost:/vol2 /mnt/glusterfs/vol-crypt
# diff /mnt/glusterfs/vol-crypt/test.txt /tmp/test.txt
#
Compromising with the same MK
73. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 73
Disk encryption
# openssl rand -hex 32 > /tmp/encryption.master-key
# diff /mnt/glusterfs/vol-crypt/test.txt /tmp/test.txt
#
Compromising with a different MK keeping mounted
74. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 74
Disk encryption
# umount /mnt/glusterfs/vol-crypt
# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/tmp/encryption.master-key
localhost:/vol2 /mnt/glusterfs/vol-crypt
# diff /mnt/glusterfs/vol-crypt/test.txt /tmp/test.txt
diff: /mnt/glusterfs/vol-crypt/test.txt: Invalid argument
# ls -lh /mnt/glusterfs/vol-crypt
total 1.1G
-rw-r--r-- 1 root root 1.0G May 18 23:31 1gb.dat
-rw-r--r-- 1 root root 32 May 18 22:57 test.dat
-rw-r--r-- 1 root root 5 May 18 22:55 test.txt
# cp /mnt/glusterfs/vol-crypt/test.txt ~/
cp: reading `/mnt/glusterfs/vol-crypt/test.txt': Invalid argument
# ls -l ~/test.txt
-rw-r--r-- 1 root root 0 May 19 00:38 /root/test.txt
Compromising with an invalid MK
75. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 75
Disk encryption
# echo "test2" > /mnt/glusterfs/vol-crypt/test2.txt
# cat /mnt/glusterfs/vol-crypt/test2.txt
test2
# diff /mnt/glusterfs/vol-crypt/test2.txt /tmp/test2.txt
#
Compromising with an invalid MK (contd.)
# rm /mnt/glusterfs/vol-crypt/test.txt
rm: cannot remove `/mnt/glusterfs/vol-crypt/test.txt': Invalid argument
# ls -lh /mnt/glusterfs/vol-crypt
total 1.1G
-rw-r--r-- 1 root root 1.0G May 18 23:31 1gb.dat
-rw-r--r-- 1 root root 6 May 19 00:39 test2.txt
-rw-r--r-- 1 root root 32 May 18 22:57 test.dat
-rw-r--r-- 1 root root 5 May 18 22:55 test.txt
# rm /mnt/glusterfs/vol-crypt/test2.txt
# ls -lh /mnt/glusterfs/vol-crypt
total 1.1G
-rw-r--r-- 1 root root 1.0G May 18 23:31 1gb.dat
-rw-r--r-- 1 root root 32 May 18 22:57 test.dat
-rw-r--r-- 1 root root 5 May 18 22:55 test.txt
Enable to write
a file with an
invalid MK.
(Is it okay?)
76. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 76
Disk encryption
# mv /mnt/glusterfs/vol-crypt/test.txt /mnt/glusterfs/vol-crypt/test2.txt
mv: cannot move `/mnt/glusterfs/vol-crypt/test.txt' to a subdirectory of itself, `/mnt/
glusterfs/vol-crypt/test2.txt'
Compromising with an invalid MK (contd.)
# umount /mnt/glusterfs/vol-crypt
# mount -t glusterfs -o xlator-option=vol2-crypt.master-key=/var/lib/glusterd/vols/vol2/
encryption.master-key localhost:/vol2 /mnt/glusterfs/vol-crypt
# ls -lh /mnt/glusterfs/vol-crypt
total 1.1G
-rw-r--r-- 1 root root 1.0G May 18 23:31 1gb.dat
-rw-r--r-- 1 root root 6 May 19 00:44 test2.txt
-rw-r--r-- 1 root root 32 May 18 22:57 test.dat
-rw-r--r-- 1 root root 5 May 18 22:55 test.txt
# cat /mnt/glusterfs/vol-crypt/test2.txt
cat: /mnt/glusterfs/vol-crypt/test2.txt: Invalid argument
# rm /mnt/glusterfs/vol-crypt/test2.txt
rm: cannot remove `/mnt/glusterfs/vol-crypt/test2.txt': Invalid argument
The proper user
cannot handle
the file created
with the invalid
MK.
(Is it okay?)
77. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 77
Disk encryption
# gluster volume info vol-crypt
Volume Name: vol2
Type: Replicate
Volume ID: e0332771-a3c2-4fe5-980c-b3860cfe3baf
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv2/vol2
Brick2: zwei:/mnt/lv2/vol2
Options Reconfigured:
encryption.data-key-size: 512
encryption.master-key: /var/lib/glusterd/vols/vol2/encryption.master-key
performance.open-behind: off
performance.write-behind: off
performance.quick-read: off
features.encryption: on
# gluster volume reset vol-crypt
volume reset: success: reset volume successful
Compromising with volume reset
78. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 78
Disk encryption
# gluster volume info vol-crypt
Volume Name: vol2
Type: Replicate
Volume ID: e0332771-a3c2-4fe5-980c-b3860cfe3baf
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: eins:/mnt/lv2/vol2
Brick2: zwei:/mnt/lv2/vol2
Compromising with volume reset (contd.)
# cat /mnt/glusterfs/vol-crypt/test2.txt
U�%U?0��x^-�bO
# cat /mnt/glusterfs/vol-crypt/test.txt
Zd��]K!q�tuv
May be a way
of cracking?
79. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 79
Enhancement
for Developers
80. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 80
A Volume
GFID Access
A Volume/.gfid
62fe0d4f-dfe9-4a2d-b811-176c6d347a7c
52fd93ea-45ba-47f0-916a-bd3774239237
e5593949-79c3-463c-909b-8cc8ef014eb4
bf758c70-ff2b-4f0d-bfc9-860ece79c246
70aaacf9-1c09-44e2-97a2-9486adf10225
e5498fc4-7345-4f5f-af59-81acff1fd083
f6a608ed-0c68-4d1a-a4d7-fb375ba8fd63
d420cbb3-c1e8-47d3-b317-0c8afbc7a8c4
a76dd563-e878-45a0-ac48-59084d86bd0c
f9dbc760-c8f3-41e6-8d24-68b24c4c577b
5c0374a8-18fe-4dd6-89e7-f6551111d980
You can deal
with each file by
GFID
Single
namespace, just
under the mount
point
81. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 81
GFID Access
# brick="/mnt/lv3/vol-gfid-access";gluster volume create vol-gfid-access eins:$brick zwei:
$brick
# gluster volume start vol-gfid-access
# mkdir /mnt/glusterfs/vol-gfid-access
# mount.glusterfs -o aux-gfid-mount localhost:/vol-gfid-access /mnt/glusterfs/vol-gfid-
access
# for i in `seq 0 9`; do dd if=/dev/zero of=/mnt/glusterfs/vol-gfid-access/$i.dat bs=1M
count=1; done
# ls -a /mnt/glusterfs/vol-gfid-access/.gfid
ls: cannot open directory /mnt/glusterfs/vol-gfid-access/.gfid: Stale file handle
# ls -a '/mnt/glusterfs/vol-gfid-access/.gfid/0svu9Cc1wVRLOBiu5NqF3ncw=='
ls: cannot access /mnt/glusterfs/vol-gfid-access/.gfid/0svu9Cc1wVRLOBiu5NqF3ncw==: No such
file or directory
# ls -ld /mnt/glusterfs/vol-gfid-access/.gfid/
drwxr-xr-x 3 root root 166 May 19 03:03 /mnt/glusterfs/vol-gfid-access/.gfid/
82. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 82
GFID Access
# stat /mnt/glusterfs/vol-gfid-access/.gfid/
File: `/mnt/glusterfs/vol-gfid-access/.gfid/'
Size: 166 Blocks: 0 IO Block: 131072 directory
Device: 16h/22d Inode: 13 Links: 3
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2014-05-19 03:03:14.146605880 +0900
Modify: 2014-05-19 03:03:04.968605874 +0900
Change: 2014-05-19 03:03:04.968605874 +0900
# strace ls -a /mnt/glusterfs/vol-gfid-access/.gfid
...
stat("/mnt/glusterfs/vol-gfid-access/.gfid", {st_mode=S_IFDIR|0755, st_size=166, ...}) = 0
open("/mnt/glusterfs/vol-gfid-access/.gfid", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = -1
ESTALE (Stale file handle)
...
• How can I let it work well?
• If it becomes to work fine, applications using GlusterFS can
manage their data in a single namespace.
83. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 83
Additional
news
84. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 84
Added files
! cli/src/cli-quoted-client
! contrib/qemu
! error-codes.json
! extras
! geo-rep
! glusterfs-georep-logrotate
! gluster-rsyslog-*.conf
! hook-scripts/add-brick
! logger.conf.example
! post-upgrade-script-for-quota.sh
! pre-upgrade-script-for-quota.sh
! geo-replication
! gf-error-codes.h.template
! libgfchangelog.pc.in
! libglusterfs/src
! client_t
! glusterfs-acl
! timespec
! rpc/rpc-lib/src/rpc-drc
! run-tests.sh
! tests
! xlators
! cluster
! dht/src
! dht-shared.c
! encryption
! crypt
! features
! changelog
! compress
! gfid-access
! glupy
! qemu-block
! quota
! quota-enforcer-client.c
! quoted-aggregator
! quoted-helpers
! performance
! readdir-ahead
! playground
! storage
! bd (replacement of
bd_map)
related qemu
codes
glupy
has merged!
a lot of test
codes!
template for
xlator
development
85. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 85
Conclusion
86. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 86
Conclusion
! Update for "everyone"
! 12 features, 8 categories.
! Contribution by HekaFS
! Disk encryption has been one of my dream since 2.0.2.
! Voice of users
! Brick Failure Detection
! Prevent NFS restart on Volume change
These are just the great community's power!
Use the latest version, and join us!
87. Copyright (C) 2014, NTTPC Communications, Inc. All Rights Reserved. 87
To contact us, e-mail here -> storage-contact@nttpc.co.jp