Life as a GlusterFS Consultant with Ivan Rossi

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Life as a consultant listed on gluster.org
Ivan Rossi (@rouge2507)
Gluster summit 2016
Ivan Rossi (@rouge2507) Life as a consultant listed on gluster.org Gluster summit 2016 1 / 18

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
What is like
What is like

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
What is like
How I ended-up there
BioDec is a micro-SME, once mainly devoted to scientific computing.
Used Gluster (on Debian) since version 3.0.
my typical use case: the multi-TB “shared folder”
“data ponds” for scientific computing apps
run small distributed or replicated Glusters
for years with no problems
(It is Justin Clift’s fault)

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
What is like
The eight-out-of-ten scenarios
The question:
“Hi there, I am John Doe of Example.com, based in Somewhere,
USA. Could you assist us on our Gluster projects?”
The (incomplete) answer:
“…We are based in Italy: support options are obviously
conditioned by our physical location…”
… connection reset by peer.
(Our location is clearly stated on the gluster.org page)

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
What is like
The 9th scenario: those looking for a scapegoat.
“Would you be able to maintain and operate Gluster for us?
Would you be able to provide 24x7 1-hour SLAs?”
(No, we can not. Red Hat can, maybe.)
Support is tricky:
What is the long-term-support, rock-solid, stable version of Gluster?
Release pace is accelerating.
Meta-quoting Jeff: “A project is not a product”
Gluster is definitely a project, RHGS is a product (I think).

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
What is like
Lessons from failures
An US-based Gluster consultancy may possibly do OK.
Stop wasting time with requests coming from the US.
Although, it would be nice to have business partners there for referral
Would Red Hat be interested in referrals?
Would it make sense to try to network between Gluster consultants?
some of you are here, tell me what happens to you.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
What is like
What kind of businesses calls you?
Small providers (VM storage for Proxmox, iSCSI).
E-commerce shops.
Publishing houses (digital news and journals) .
E-learning shops.
Security/surveillance (digital video).
Plus a mixed-bunch that is misusing Gluster.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
What is like
Who actually did business with you?
(Those looking for a Joe-Julian impersonator, after he turned them down)
People looking for a review of their configuration/install
People that hurt themselves using Gluster.
People that hurt themselves using Gluster-in-the-sky.
Scared people that do not want to hurt, but that cannot call Red Hat.
People not looking for Gluster, but that get it anyway.
Some of our own clients.
Cheapskates.
Notice: nobody that has my once-typical use case ever calls me.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Stories
Stories

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Stories
The sleepy ls
My only business success with a US-based company.
E-learning company
Hardware-based 3.5.x D-R setup, 60 TB, geo-replicated to UK.
Mainly serving static content (videos) plus some wordpress
“Happy with it, but recently it became slow-responsive: ‘ls’ takes
order of minutes to start listing”
Root cause
Millions of files ALL in the root of the volume. No subdirs at all.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Stories
The Gluster key-value store
“We use Gluster to store many small-files generated by legacy AS-400
applications. The files are 1k long at most. We would like to speed-up
Gluster”
(What about getting PostgreSQL? or Riak?)
“We are assessing Gluster reliability and we are not happy. We write 1k
files to a R3 cluster. Peers also run processing apps and Nginx. We read
directly from the bricks for performance reasons. If we pull the plug on a
server, Gluster ends-up inconsistent. We think about replacing it with
MongoDB”
(Be my guest. Anyway, you are running without quorums)

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Stories
Gluster in the sky (with diamonds)
Everyone is reaching for the Clouds (is this the “digital transformation”?).
Then many discover that they want/need “the NetApp in the sky”.
Gluster as a replacement NetApp-in-the-clouds
No real competitor.
Lustre, BeeGFS have different use cases.
LizardFS/MooseFS depends on multicast for HA.
Amazon’s EFS??
RH not providing support for THAT cloud
Fear of scaling-out license costs (RHGS)
Gluster has not been designed (from scratch) to be “cloud native”.

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Stories
GSD: the placid e-commerce
British e-commerce, web stack running on a Vmware provider.
Two 3.6.x R3 cluster on two DC, one the geo-replica of the other,
containing the static content to be served (images and PDFs).
Maximum bandwidth = 200 Mbit (guaranteed)
It is almost OK (to my surprise)
Small volumes, few MB-writes per day from a single FTP entry point.
Varnish and Nginx effectively shield Gluster from the traffic.
Geo-replication issue: by copying a directory tree and then doing a
rm-mv sequence, they can consistently split-brain the geo-replicated
slave (bugzilla: 1154307, persisting on 3.7.4).

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Stories
GSD: “Immutable infrastructure” Glusters on AWS
British publishing house (journals)
A single R3 3.6.x cluster at the core of the web stack
20+ volumes w 1 EBS each (no lvm), one 3TB “problematic” volume
Default “quickstart-type” config.
Full heal or a resync lasts two weeks. They want to create new
(populated) clusters by cloning the EBS volumes of the bricks and by
creating new clusters on top of them.
This way they want to do upgrades, disposable systems for devs, disaster
recovery…

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Stories
GSD: the undying brick
Italian major publishing house.
Several Gluster clusters (mostly R2) on Azure. Building full-featured
R3 to host their main news sites (3.7.13).
Configuration looks nice, minor tweaks suggested.
The incidents
on network partition, gluster loses server quorum, fencing a peer.
on reconnection: the peer has lost the vdisk hosting the bricks, XFS
filesystem umounts, but Gluster recreates the brick/ folder within the
empty mount-point path and self-heal floods the root partition
reproduced using iSCSI
bugzilla: 1378978

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Stories
Our-own “hyperconverged” SciComp data center
German startup doing genomics.
Three servers: 20-core, 256 GB RAM, 1 NVME, 8 3TB SAS disks.
2xGBE + 2x10GBE on copper.
Debian 8 + Ganeti + Gluster 3.7.15.
Ganeti will manage the KVM VMs, all of them connected to both GBE
and 10GBE. Gluster will provide the “multi-TB shared folder”. Gluster
peers will either be the hosts (cgroups) or LXC containers. Still not
decided if do R3 or 2+1 dispersed. Advice welcome.
(Why not use Ovirt? Never found the occasion to study it properly)

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Stories
Thanks
Niels de Vos
Joe Julian
Vijay Bellur
Jeff Darcy
the IRC people
(and to you, for bearing with me)

.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Stories
Extreme tech-support on #gluster IRC
xxxxx - Hi experts, I setup a three node gluster replication, but
the sync time between nodes are too long (10 min) (I wrote
directly from filesystem without mounting the glusterFS via this
command: “mount -t glusterfs glustername:/volumename
mountpoint”). How can improve the time to 1 min?
atinm - xxxxx: why aren’t you writing from mount point?
xxxxx - atinm: because r/w speed decrease to network limit
speed
atinm - if you don’t use the mount point, how are you leveraging
the distributed storage here?
xxxxx - atinm: i r/w directly to partition “like /dev/sdc1”

Life as a GlusterFS Consultant with Ivan Rossi

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (10)

Similar to Life as a GlusterFS Consultant with Ivan Rossi

Similar to Life as a GlusterFS Consultant with Ivan Rossi (20)

More from Gluster.org

More from Gluster.org (20)

Recently uploaded

Recently uploaded (20)

Life as a GlusterFS Consultant with Ivan Rossi