Linux provide facilities to expose emulated LUNs to initiators using Linux-IO (LIO) scsi target implementation . LIO not only support exposing conventional block devices but also supports other storage interfaces like file or memory based LUNs. Also it supports multiple fabric interfaces - FC, FCoE, iscsi and many more. LIO can be used in SAN environments with minimal storage resources. Native support for LIO in linux hypervisors and in Openstack make it a good storage option for cloud deployments. This presentation includes demo slides with LIO iscsi target implementation.

1. 1

2. Agenda
• SCSI Targets
• STGT and Need of LIO
• LIO Introduction
• LIO Architecture
• LIO Directory structure
• Linux Storage Stack and LIO
• LIO Configfs Interface
• LIO Configfs tree
• LIO Command Line interface
• Use cases
• LIO users and partners
• Demo
2

3. SCSI targets
• The subsystem enables a computer node to behave as
a SCSI storage device, responding to storage requests
by other SCSI initiator nodes.
• Opens up the possibility of creating custom SCSI
devices and putting intelligence behind the storage.
• Extra functionality can be added on top of it – RAID,
snapshots, replication, HA, backup, data deduplication.
• Some existing SCSI target implementations – SCST, IET,
STGT
3

4. STGT and Need of LIO
• Many major distributions included STGT support
for userspace iSCSI target mode
• But is still unsupported in many commercial cases
• Major distributors wanted to see proper in-kernel
iSCSI target mode support and the community
offered out of tree modules for those interested
• Many developers, hardware vendors and users
wanted upstream kernel level iSCSI target mode
4

5. LIO Introduction
• TCM/LIO = Target core module / Linux- IO
• Open source Linux SCSI target
• Developed by Datera Inc. (Rising tide systems)
• Merged in Linux Kernel on March, 2011
• Requirements
o Linux Kernel version >= 2.6.38
• Supported by almost all Linux distributions by
default
5

6. LIO Introduction…Contd
• Supports SCSI Advanced features
o Persistent Reservations (SPC3/SPC4)
o ALUA (Asymmetric Logical Unit Assignment)
Both explicit and implicit
o T10 Data Integrity Format
o VAAI (vStorage APIs for Array Integration)
o Multiple connection per session (MC/S)
o iSCSI ERLs (Error Recovery Levels)
ERL 0 – Session Recovery
ERL 1 – Digest failure Recovery
ERL 2 – Connection Recovery
6

7. LIO Architecture
7

8. LIO Architecture…Contd
LIO consists of three group of modules :
• Fabric modules
o Frontend of LIO, encapsulates supported storage protocols
o Supports iSCSI, FC, FCOE, SRP, infiniband, vHost
• Storage modules / backstores
o Backend of LIO – provide access to imported devices via device drivers
o Supports –
Block ( HDDs, SSDs, LVM, Raids )
File
RAMdisk
• Core Functionality (target_core_mod.ko)
o Manages resources and memory
o Implements Persistent resvervation, ALUA and other core features.
8

9. Linux Storage Stack and LIO
9
VFS (Virtual File System)
Memory Management
ext3 ext4 others
Block Layer and Device Mapper
SCSI Layer
target
/dev/sdb
LIO core modules
FC iSCSI SRP
Block Fileio Raw
/dev/mapper/xxx /dev/vg/lv ext4
file
Logical Volume Manager
Low Level
drivers
user
Kernel
Storage

10. LIO directory Structure
10
Storage and
core modules
Core data
structures
declarations
LIO dir
structure

11. LIO Configfs Interface
• Configfs is a RAM-based virtual file system similar
to sysfs (but not same)
• Precisely, it is inverse of sysfs.
• Provide better control from userland
• Unlike sysfs, configfs allows to create, manage
and destroy kernel objects from user-space
• Typically mounted at /config or /sys/kernel/config
• LIO uses it for management and configuration of
storage and fabric modules.
11

12. LIO Configfs tree
12

13. LIO Command Line Interface
• lio_node and tcm_node
o Low level utility to manage storage and fabric configfs nodes
o Python based cli for developers, intergrators and advanced users
o Different tcm utilities available for fabric modules
• lio-dump, tcm-dump
o Save configuration
• target-cli
o Provide unified single node SAN configuration shell
o Fabric agnostic
o Exports functionality via RTSlib library and API
python-rtslib (python API library over configfs)
13

14. Use Cases
• HA clustering
• Openstack Cinder apis written to consume LIO
iSCSI storage
• IO virtualization(IOV) with KVM on IOV capable
hardware
• iSCSI LAN and WAN replication – DRBD and
pacemaker support
• iSCSI boot using PXE
• Implementing advanced features on top of LIO –
dedupe, thin provisioning, etc
14

15. LIO users and partners
15

16. Its DEMO time
16

17. iSCSI discovery and LIO
Data structures
17
STEP 2:
# Create 2 sparse files and we will export them as file based LUNs
dd if=/dev/zero of=/data/file1 bs=1K count=1 seek=10000
dd if=/dev/zero of=/data/file2 bs=1K count=1 seek=10000
file1
file2
STEP 1:
# Start the target rc script
# This will insert target core module (core Engine activated)
/etc/init.d/target start

18. iSCSI discovery and LIO
Data structures
18
fileio_0
disk1
disk2
default_tg_
pt_gp
default_tg_
pt_gp
STEP 3:
# Allocate fd_dev and create file based luns
# This will insert LIO’s fileio storage module
tcm_node --fileio fileio_0/disk1 /data/file1 10240000
tcm_node --fileio fileio_0/disk2 /data/file2 10240000
file1(fd_dev)
file2 (fd_dev)
se_hba
se_device
t10_alua_tg_
pt_gp

19. iSCSI discovery and LIO
Data structures
19
fileio_0
disk1
disk2
default_tg_
pt_gp
default_tg_
pt_gp
STEP 4:
# Add and bind LUN0 to 1st target port of portal group 1
lio_node --addlun <target_iqn1> 1 0 iscsi00 fileio_0/disk1
# Add and bind LUN0 to 2nd target port of portal group 1
lio_node --addlun <target_iqn2> 1 0 iscsi00 fileio_0/disk1
# Add and bind LUN1 to 1st target port of portal group 1
lio_node --addlun <target_iqn1> 1 1 iscsi01 fileio_0/disk2
# Add and bind LUN1 to 2nd target port of portal group 1
lio_node --addlun <target_iqn2> 1 1 iscsi01 fileio_0/disk2
file1(fd_dev)
file2 (fd_dev)
se_hba
se_device
t10_alua_tg_
pt_gp
tpgt_1
lun_0
lun_1
se_lun
iscsi_portal
_group

20. iSCSI discovery – Target port binding
20
TCM/LIO
file1 file2
lun 0
(iscisi00)
lun 1
(iscsi01)
target_port0 target_port1
==TARGET PORT BINDING==
lun0 binds to target_port0 and target_port1
lun1 binds to target_port0 and target_port1

21. iSCSI discovery and LIO
Data structures
21
fileio_0
disk1
disk2
default_tg_
pt_gp
default_tg_
pt_gp
file1(fd_dev)
file2 (fd_dev)
se_hba
se_device
t10_alua_tg_
pt_gp
tpgt_1
lun_0
lun_1
se_lun
iscsi_portal
_group
STEP 5:
# Create network portals for target ports
lio_node --addnp <target_iqn1> 1 <IP address:port>
lio_node --addnp <target_iqn2> 1 <IP address:port>
IP:port
IP:port
iscsi_np
iscsi_np
STEP 6:
# Disable iSCSI authentication of target portal group . Enabled by default
lio_node --disableauth <target_iqn1> 1
lio_node --disableauth <target_iqn2> 1

22. iSCSI discovery and LIO
Data structures
22
fileio_0
disk1
disk2
default_tg_
pt_gp
default_tg_
pt_gp
file1(fd_dev)
file2 (fd_dev)
se_hba
se_device
t10_alua_tg_
pt_gp
tpgt_1
lun_0
lun_1
se_lun
iscsi_portal
_group
STEP 7:
# Give access of bound luns to initiator and assign mapped lun numbers
lio_node –addlunacl <target_iqn1> 1 <initiator_iqn> 0 0
lio_node --addlunacl <target_iqn2> 1 <initiator_iqn> 0 1
lio_node --addlunacl <target_iqn1> 1 <initiator_iqn> 1 2
lio_node --addlunacl <target_iqn2> 1 <initiator_iqn> 1 3
IP:port
IP:port
iscsi_np
iscsi_np
lun_0 lun_2
lun_1 lun_3
Mapped
luns
STEP 8:
# Enable target portal group
lio_node --enabletpg <target_iqn1> 1
lio_node --enabletpg <target_iqn2> 1

23. 23
TCM/LIO
file1 file2
lun 0
(iscisi00)
lun 1
(iscsi01)
target_port1 target_port2
IP:Port IP:Port
Network portals Target portal
group
iSCSI discovery – Network Portals and
Target Portal Group
lun 0
lun 1 lun 2
lun 3

24. lnitiator
iSCSI discovery and LIO
Data structures
24
fileio_0
disk1
disk2
default_tg_
pt_gp
default_tg_
pt_gp
file1(fd_dev)
file2 (fd_dev)
se_hba
se_device
t10_alua_tg_
pt_gp
tpgt_1
lun_0
lun_1
se_lun
iscsi_portal
_group
STEP 9:
# Initiator discovers and login to target ports to establish iscsi session with target
iscsiadm --mode discovery -t sendtargets --portal <IPaddr:Port>
iscsiadm --mode node --targetname <target_iqn1> --portal <IPaddr:Port> --login
iscsiadm --mode node --targetname <target_iqn2> --portal <IPaddr:Port> --login
IP:port
IP:port
iscsi_np
iscsi_np
lun_0 lun_2
lun_1 lun_3
Mapped
luns

25. iSCSI login
and session
creation
25
TCM/LIO
file1 file2
lun 0
(iscisi00)
lun 1
(iscsi01)
target_port1 target_port2
IP:Port IP:Port
Network portals Target portal
group
/dev/sda
/dev/sdb
/dev/sdc
/dev/sdd
Initiator
lun 0
lun 1 lun 2
lun 3

26. References
• RFC 3720
• LIO wiki page
• SCST vs LIO
• iscsi targets wiki page
• LIO-utils – github
• Configfs
• Targetcli and rtslib
• Cinder plugin for LIO
• Open iSCSI utility
• STGT project
• Distributed replicated block device
26

27. Questions ?
27

28. Author Biography
28
Kashish Bhatia – Member of Technical Staff, VMware
• Technology enthusiast, with most of the experience in storage domain
• Has exposure to other Linux subsystems like filesystems, block layer, Selinux
• Experience of working on distributed file system (Lustre) and storage virtualization
• Interests : open source, linux kernel hacking
• Published technical paper in Openstack Summit Nov 2013
• Bachelor of Engineering, Computer Science, Pune University
• To know more about Kashish connect on LinkedIn