1) The document discusses using iSCSI with Ceph storage to provide block storage access in addition to Ceph's native object and file storage capabilities. 2) It describes EA's implementation using Linux target (LIO) iSCSI gateways to connect StorNext hosts and backup hosts to a Ceph storage cluster. 3) While performance of up to 1GB/s was achieved, current configurations are resource constrained and LIO faults are difficult to recover from, with the main lessons being around effective resource management in the Ceph and iSCSI configuration.

1. ADVENTURES IN iSCSI
Paul Evans
principal architect
daystrom technology group
paul at daystrom dot com
los angeles
ceph day
SQUEEZING WAY-COOL STORAGE
TECH INTO A LEGACY MODEL
+
Tu Holmes
storage lead
electronic arts
tu at ea dot com

2. WHAT’S IN THIS TALK
• Why (add) iSCSI?
• the How of iSCSI + Ceph
• the Good & Bad
• Lessons (hopefully) Learned

3. CEPH: THE FUTURE OF STORAGEWHY SADDLE IT WITH OLD (SCSI) TECH?

4. Reasons for iSCSI
IncompatibilitiesVirtualization Databases

5. EA iSCSI REASONS…
✓ Kernel
(and Desire to use VolMgr)
Incompatibility

6. CEPH+ISCSI
x12 x12 x12 x12 x12 x12
OSD Host MON Host iSCSI GTWY Host
Main Network
H H H H

7. DESIRED iSCSI FEATURES
MaximumThroughput
Good Support
HA Options

8. CEPH+iSCSI OPTIONS
LIO
(targetcli) SCST TGT iET
Maintainer
Datera (rt)
Red Hat (fb)
SCST Ltd. Community
(FUJITATomonori)
Community
Lastest Stable
2.1.0 (rt)
2.1.fb41(fb)
3.0.1 1.0.60 1.4.20
Latest
Commit
Jan 2015 (rt)
Jun 2015 (fb)
June 2015 Jul 2015 Jun 2014
Next Release 3.0 (?) 3.1 ? ?
Mainline
Kernel Yes No No No

9. Suppported Features
LIO SCST STGT iET
Kernel Yes Yes User Space Split
RBD Yes Yes Yes No
iSER Yes Pre-Release Yes No
SRP (IB) Yes Yes No No
ALUA Yes Yes No No

10. EACLUSTER
CURRENT OSD Service
MON Service
iSCSI GTWY Service
H StorNext Host
StorNext Backup Host
Vol Mgr
x12 x12 x12 x12 x12 x12 x12 x12
H H H HH H

11. EACLUSTER
POOLCONFIG
CEPH
iSCSI
VolMgrBACKUP HOST
ISCSI GATEWAYS
CEPH CLUSTER
ECTier
CacheTier
RBD RBD RBD RBD RBD RBD

12. HWSPECS
SCALE-OUT CLUSTER OF STORAGE
OBJECT+FILE+BLOCK ACCESS
HIGH AVAILABILITY
SELF-HEALING
SIMPLELOW-COST
SCALE-OUTSTORAGE
THE SIMPLE SOLUTION FOR
THE STORAGE OF EVERYTHING
The storageFOUNDRY Nautilus system is the smart choice for
organizations with diverse storage needs and explosive data growth.
Nautilus is a massively scalable, open-source storage solution developed from
the ground-up to provide object, block and file-system access in one self-
managing, self-healing platform. Using a reliable non-stop architecture, this
multi-streamed approach to storage grids is the flexible solution for Data
Automation, High-Throughput Media or Analytics, and OpenStack Clouds.
The Nautilus C100 series are fast and dense 1U storage nodes for delivering
from 48TB to 120TB in a single rack unit, with a low-service architecture to
minimize in-field support needs. The C100 nodes provide a range of protocols
and speeds, from 1G Ethernet to 56G InfiniBand, enabling it to create stand-
alone C-series Cloud deployments, or drop into applicationtuned roles for 4K-
Media, GS-Genomics and E-series deployments.
Nautilus answers the call for multi-protocol, cost-effective and scalable
storage that serves up classic Enterprise Storage I/O and well as innovative
direct-access data, making it the simple choice for today’s unpredictable
computing needs and THE STORAGE OF EVERYTHING.
NFS
SMB
FTP
HTTP
iSCSI
1/10/40G ETH
56G InfiniBand
8G/16G FC
C100/C140/C150
SCALE-OUT STORAGE NODES
5600NS
NAUTILUS
NAUTILUSobject storage
8x C100-48T 1U Nodes
• 12 x 4TB SATA HDD
• 1 x 400G SATA SSD
• 1 x E3-1200 4Core CPU
• 32G ECC DRAM
• 1 x 10GE / 4 x 1GE
foundry
storage
www.storageFOUNDRY.net
Nautilus C100-G3
Nautilus C300-G3
Nautilus 4K

13. GOOD&BAD ✓ Up to 1GB/sec Burst
✓ 600MB Sustained Write
Performance
Scalability
Availability
ReliabilityB
✓ Add cluster (Ceph) storage on-the-fly
✓ Dynamically Expand/Add LUNs
✓ Ceph is naturally Highly Available
✓ iSCSI HA via Active/Passive Gateways
✓ Current config: resource constrained
✓ Difficult to recover from LIO faults

14. RELIABILITY
✓ Limit Scrubbing to Low-LoadTimes
✓ Limit Background Maintenance/Repair
✓ Limit PGs to maximize resources
✓ Monitor/Track occurrences of Blocked IO
ReliabilityB
Tunings

15. EACLUSTER NEXT-GEN
x12 x12 x12 x12 x12 x12 x12 x12
OSD Service
MON Service
iSCSI GTWY Service
H StorNext Host
StorNext Backup Host
Vol Mgr
H H H HH H

16. INTOTHEFUTURE
Active-Active (i)SCSI Support is comingI
➡ Updates to both LIO & Ceph occurring
➡ ‘Support’ forVMware / Persistent Reservations
➡ Timeline: 60-180 days…YMMV
Futures…

17. theThree
Lessons
Learned
Resource Management
Resource Management
Resource Management
It’s mostly about…

18. thank you!
Paul Evans
principal architect
paul at daystrom dot com
technology grouptechnology group
Tu Holmes
Storage Lead
tu at ea dot com