Robert Sander, Heinlein Support GmbH

1. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
and iSCSI
in a
High Availability Setup

2. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Who?
➞ 20 years experience and
knowledge around Linux servers
and Mail
➞ IT Consulting and 24/7 Linux-
Support with > 20 employees
➞ Incorporated ISP since 1992
➞ jpberlin.de
➞ mailbox.org
➞ Daily insights into the IT of
small, medium and large
businesses

3. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Goal:
Connecting
Classic and Modern
Storage Concepts
+

4. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Intro:
A quick peek on the technologies involved

5. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Ceph

6. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
iSCSI

7. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
A peek on iSCSI
➞ Easy to implement
➞ Software-defined block storage
➞ established failover mechanisms
➞ ALUA
➞ and proprietary path priorisations (EMC,
NetApp, …)

8. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Why ceph + iSCSI?
➞ Modern and classic storage
concepts combined
➞ Not every platform supports Rados Block
Devices (RBD)
➞ iSCSI implementations widely available
➞ iSCSI Targets usually not implemented
over portals of several hosts
➞ High Availability only available in
hardware
➞ using a shared backplane
➞ Linux TGT has native RBD support in
userland
➞ needs bs_rbd
➞ in mainline since 02/2013

9. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Implementation
Test

10. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Prerequisites
➞ Running Ceph Cluster
➞ Tested with Ceph Firefly 0.80.5 and Ceph
Giant 0.87
➞ Two or more hosts as ceph ↔
iSCSI „Bridges“
➞ Two NICs for Backing Store (ceph) and
iSCSI Portal
➞ Tested on Ubuntu 14.04 LTS / ceph 0.80.5
as well as ceph 0.87 and tgt 1.0.43
➞ Hosts / Cluster / Applications
➞ with iSCSI support

11. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Concept
➞ Ceph accesses objects in
parallel on OSDs
➞ Example with multiple RadosGWs and

12. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Concept
➞ Parallel access also works with
TGT iSCSI target

13. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Preparation of Ceph iSCSI bridge↔
➞ Installation on Ubuntu 14.04 LTS
➞ Connection to the Ceph cluster
(configuration)
tgt1 # aptitude install ceph tgt
tgt1 # cat /etc/ceph/ceph.conf
[global]
mon_host = 192.168.1.247,192.168.1.248,192.168.1.249
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
[client]
rbd_cache = false
# needed on the iSCSI "bridge" hosts

14. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Preparation of Ceph iSCSI bridge↔
➞ Connection to the Ceph cluster
(authentication)
➞ Better: dedicated account for target pool
via „ceph auth“
tgt1 # cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
key = HeinleinSupportHeinleinSupportHeinlein==

15. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Preparation of Ceph iSCSI bridge↔
➞ Create a Ceph RBD as base for
iSCSI LUN
➞ Example: a 3 GB sized RBD „ceph-tgt-shared“.
➞ As usual the storage space is not
allocated but thin-provisioned
➞ The RBD will not be used with the kernel driver
and does not get mapped
➞ Reminder: RBD caching has to be deactived
tgt1 # rbd create ­­size 3072 ceph­tgt­shared
tgt1 # rbd info ceph­tgt­shared
rbd image 'ceph­tgt­shared':
size 3072 MB in 768 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.56c7.238e1f29
format: 1

16. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Preparation of Ceph iSCSI bridge↔
➞ Create new iSCSI Portal
➞ IQN structure: iqn.YYYY-
MM.NAMING_AUTHORITY:UNIQUE_NAME
➞ create LUN #1 with RBD as
storage backend
➞ bind portal and target
tgt1 # tgtadm ­­lld iscsi ­­op new ­­mode target
­­tid 1
­T iqn.2005­10.de.heinlein­support:storage.s0.ceph.shared.tgt
tgt1 # tgtadm ­­lld iscsi ­­mode logicalunit ­­op new
­­tid 1 ­­lun 1 ­­bstype rbd ­­backing­store ceph­tgt­shared
­­bsopts "conf=/etc/ceph/ceph.conf;id=admin"
tgt1 # tgtadm ­­lld iscsi ­­op bind ­­mode target ­­tid 1
­I ALL

17. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Preparation of Ceph iSCSI bridge↔
➞ Repeat Ceph and TGT setup on
additional „bridge“ hosts
➞ ! tgt configuration cannot (as of version
1.1x) be created
completely by tgt-admin
➞ The usual way:
tgt-admin –dump > /etc/tgt/conf.d/targets.conf
does no contain parameters bstype and
bsopts.
➞ Write target configuration manually

18. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Preparation of Ceph iSCSI bridge↔
➞ Write target configuration
➞ Create configuration identical on all
„bridge“ hosts
➞ Right now we have 2+ iSCSI
targets with one Ceph RBD as LUN
➞ Storage part completed :)
tgt1 # cat /etc/tgt/conf.d/ceph­tgt­shared.conf
default­driver iscsi # better located in /etc/tgt/target.conf
<target iqn.2005­10.de.heinlein­support:storage.s0.ceph.shared.tgt>
driver iscsi
bs­type rbd
backing­store rbd/ceph­tgt­shared
</target>

19. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Using the iSCSI targets
➞ Installation on Ubuntu 14.04 LTS
➞ Discover iSCSI targets
➞ Repeat for every iSCSI „bridge“ host
➞ Quick Check.
Every iSCSI host should be listed with IP
and identical IQN
client # apt­get install open­iscsi multipath­tools
client # service open­iscsi start
client # service multipath­tools start
client # iscsiadm ­m discoverydb ­t sendtargets
­­portal 192.168.100.88 ­­discover
client # iscsiadm ­m node
192.168.100.88:3260,1 iqn.2005­10.de.heinlein­support:st..
192.168.100.91:3260,1 iqn.2005­10.de.heinlein­support:st..

20. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Multipath Configuration
➞ Option 1: focus on performance
➞ Alternativly every WWID separate in
multipaths { multipath { ...
client # cat /etc/multipath.conf
defaults {
user_friendly_names yes
polling_interval 2
path_grouping_policy group_by_serial
features "1 queue_if_no_path"
path_checker directio
rr_min_io 100
failback immediate
no_path_retry 5 # 5 retries, evtl. „queue“
}
[...]

21. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Multipath Configuration
➞ Option 2: focus on availability
➞ Alternativly every WWID separate in
multipaths { multipath { ...
client # cat /etc/multipath.conf
defaults {
user_friendly_names yes
polling_interval 2
path_grouping_policy failover
features "1 queue_if_no_path"
path_checker directio
rr_min_io 100
failback immediate
no_path_retry fail # evtl. 5 (retries)
}
[...]

22. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Multipath Configuration and iSCSI Login
➞ Activate Multipath
configuration…
➞ Or: restart multipath-tools
➞ … and login to iSCSI Portals
➞ login should be successful for every
portal, i.e. every iSCSI „bridge“ host
➞ Done!
client # echo reconfigure | multipathd ­k
client # iscsiadm ­m node ­­login
Logging in to [iface: default, target: iqn.2005­10.de.he..
Logging in to [iface: default, target: iqn.2005­10.de.he..
Login to [iface: default, target: iqn.2005­10.de.heinlei..
Login to [iface: default, target: iqn.2005­10.de.heinlei..

23. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Concluding Test
➞ Check MP configuration of iSCSI
LUN
➞ Depending on /etc/multipath.conf a device file
/dev/mapper/mpath0 or
➞ the alias /dev/mapper/ceph­tgt­shared exist
➞ Finito!
client # multipath ­ll
ceph­tgt­shared (33000000100000001) dm­3 IET,VIRTUAL­DISK
size=3.0G features='1 queue_if_no_path' hwhandler='0'
wp=rw
`­+­ policy='round­robin 0' prio=1 status=active
|­ 28:0:0:1 sdf 8:80 active ready running
`­ 29:0:0:1 sdg 8:96 active ready running

24. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Another class of Initiators
Hypervisors...
➞ Successful test with iSCSI-
Initiator on XenServer 6.5
➞ [x] activate Multipath (via XenCenter)
➞ proceed with Ubuntu 14.04 LTS steps:
➞ Discover all Portals via iscsiadm
➞ Extend /etc/multipath.conf with Device Vendor
„IET*“ and Product „VIRTUAL*“. Add
path_grouping_policy group_by_serial or failover.
➞ restart or reconfigure multipathd
➞ Add new Storage Repository (SR) via one
portal
➞ The generated description of the SR
shows only one portal
➞ Looking at the CLI reveals, that
multipath is used with all portals
➞ It can be necessary to login to all
portals when starting with the first SR:

25. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Final Considerations
➞ Linux TGT is not limited to iSCSI
(SCSI over IP)
➞ RBD should be exportable via FCoE or iSER
(via RDMA)
➞ Export via FC-HBA needs to be tested
➞ Resizing?!?
➞ Not trivial
➞ resize RBD
➞ stop all portals
➞ start only one portal, then the rest
➞ Reread portal LUN with iscsiadm or
„discover“ as forced method
➞ iscsiadm „logout“ and „login“
➞ Restart multipathd or „reconfigure“
➞ resize content of LUN (pv, partition,

26. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Final Considerations
➞ Kernel RBD is more performant
➞ But causes split brain and inconsistencies
with multipathing
➞ tgt shows adequate performance in
userland
➞ I/O caches have to be deactivated
➞ Kernel RBD loses performance gain
➞ cannot be deactivated completely
➞ LIO is more current target
framework
➞ stgt runs 100% in userland, LIO is based in
kernel
➞ Mike Christie / RedHat develop on LIO
framework
➞ Data synchronisation is currently
unsolved for PREEMPT/ABORT, ALUA-
Status, etc.

27. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Performance!
24 OSD – 3 Nodes – 3 Journal SSD
2 iSCSI Bridge Nodes
10 GBit/s Ceph – 10 GBit/s iSCSI
XenServer 6.5 – Ubuntu 14.04 VM
iozone3 in one VM

28. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>

29. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>

30. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>

31. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>

32. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>

33. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>

34. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
References
➞ ceph – Storage Cluster Quick
Start ( ceph-deploy )
➞ http://ceph.com/docs/master/start/quick-ceph
➞ Linux SCSI target framework
➞ http://stgt.sourceforge.net/
➞ Generic SCSI Target Subsystem
for Linux
➞ http://scst.sourceforge.net/
➞ Multipath I/O
➞ http://en.wikipedia.org/wiki/Multipath_I/O
➞ http://en.wikipedia.org/wiki/Linux_DM_Multipath
➞ /usr/share/doc/multipath-
tools/examples/multipath.conf.annotated.

35. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
➞ We are here to help you:
➞ Robert Sander
➞ Mail: r.sander@heinlein-support.de
➞ Telefon: 030/40 50 51 – 43
➞ In case of emergency:
➞ Heinlein Support 24/7 Notfall-Hotline:
030/40 505 - 110

36. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
OpenStack DACH Day 2015

June 11, 2015

Urania Berlin

www.openstack-dach.org

37. Linux höchstpersönlich.
Highly Available iSCSI
Storage with Ceph
Stephan Seitz
<s.seitz@heinlein-
support.de>
Robert Sander
<r.sander@heinlein-
support.de>
Heinlein Support hilft bei
allen Fragen rund um Linux-Server
HEINLEIN AKADEMIE
Von Profis für Profis:
Wir vermitteln die
oberen 10% Wissen:
geballtes Wissen und
umfang-reiche
Praxiserfahrung.
HEINLEIN CONSULTING
Das Backup für Ihre
Linux-Administration:
LPIC-2-Profis lösen im
CompetenceCall
Notfälle, auch in SLAs
mit 24/7-Verfügbarkeit.
HEINLEIN HOSTING
Individuelles Business-
Hosting mit perfekter
Maintenance durch
unsere Profis.
Sicherheit und
Verfügbarkeit stehen
an erster Stelle.
HEINLEIN ELEMENTS
Hard- und Software-
Appliances und
speziell für den
Serverbetrieb
konzipierte Software
rund ums Thema eMail.