Gluster can provide block storage using LIO/TCMU. It was demonstrated providing an iSCSI block device from a Gluster volume, including block snapshots. Performance numbers were shown and it can integrate with containers by providing a persistent block device. Kubernetes was demonstrated using Gluster block storage by having nodes initiate iSCSI sessions to access a target device mounted in pods. Future work may include more testing, Heketi integration for provisioning, and hyper-convergence.

1. GLUSTER AS BLOCK STORAGE
GLUSTER DEVELOPER SUMMIT
#IRC
PKALEVER
OCT-07-2016

2. Block storage Architecture
​LIO/TCMU
TCMU-Runner
Targetcli
Demo of block store
Block Snapshots
Performance Numbers
Integration with containers
Persistent store in
Containers
Demo with k8s
Read Write Once
What's Next ?
Q&A
INDEX

3. looplback
ARCHITECTURE
iSCSI Initiator
LIO core
tcmu-runner
tcm-user
fileIO
ramdisk
block
pscsi
vhost
fabrics
storage modules / Backstores
targetcli
uio0 uio1
qcow2 glfs
/dev/sda
User
Kernel
iSCSI

4. DEMO

5. BLOCK SNAPSHOTS
Change in tcmu-runner qcow2 handler
Reflinks based snapshots
QEMU Translator from Gluster

6. BLOCK SNAPSHOTS [CONT...]
refcountheader
table
L1refcount
block
L2 datadata datadata L1 L2snap
1 2 3 N
Qemu Block Xlator
TCMU Runner
LIO/TCMU
/dev/sda
glfs qcow2
Server
Client
User
Kernel
raw
qcow2

7. PERFORMANCE
more

8. READ PERFORMANCE

9. WRITE PERFORMANCE

10. “ Containers isolate application from Env
“ Persistent storage
“ iSCSI Targer device
“ File in Gluster volume
CONTAINERIZATION GOAL
“ They are stateless

11. KUBERNETES ARCHITECTURE
“ All the kubernetes nodes initiate the iSCSI session, attaches iSCSI target
as block device and serve it to Kubernetes pod where the application is
running and requires persistent storage.

12. DEMO PLAN
From Node 6 create the pod and examine the iSCSI target device mount and
works as expected
We use 6 nodes (VMs) [ 3S for gluster & target side and other 3C for k8s Cluster & Initiator]
We create a gluster replica 3 volume using the 3 nodes {Node1, Node2 and Node3}.
Define iSCSI target using the same nodes, expose ‘LUN’ from each of them.
Use Node 4 and Node 5 as iSCSI initiators, by logging-in to the iSCSI target
session created above
Setup K8s cluster by using {Node4, Node5 and Node6}, Node 6 is master and
other 2 are slave nodes

13. DEMO
Demo on setting up multiple target with same wwn
https://asciinema.org/a/88325

14. READ WRITE ONCE
PERSISTENT RESERVATIONS OR PGR
IO fencing:
SCSI-3 PGR's are most commonly employed as fencing mechanisms in
High-availability clusters to safegaurd against split-brain conditions.
Implemented in LIO target engine
Logical mechanism for controlling the access to a device server/target

15. READ WRITE ONCE [CONT...]
HOW THEY WORK ?
PERSISTENT RESERVE IN and PERSISTENT RESERVE OUT SCSI commands.
PROUT (Service Actions)
REGISTER, RESERVE, RELEASE ..
PROUT (Reservation Types)
Write Exclusive, Exclusive Access, Write Exclusive Registrants Only,...
PRIN (Service Actions)
READ_KEYS, READ_RESERVATION, REPORT_CAPABILITIES, READ_FULL_STATUS
Example:
# sg_persist --out --register --param-rk=0x12 --prout-type=6 /dev/sda
# sg_persist --read-reservation /dev/sda

16. INTEGRATION WITH CONTAINERS
OS Level Virtualization
Docker Orchestration
PaaS Platform on top of K8s
Kubernetes
Hint: Click on the logos to navigate to respective blogs

17. WHAT'S NEXT?
1 More Functional testing with various workloads
2 Performance with container workloads
3 Gdeploy Integration
4 Dynamic Provisioning with Heketi
5 Hyper-Convergence ?
6 Thoughts are welcome!

18. Q & A