In this Joyent Technical Discussion, software engineer Jordan Hendricks discusses adding NVMe hotplug support to illumos. The project covers both coordinated hotplug and surprise removal of NVMe SSDs. In the talk, Jordan discusses the project scope, some implementation challenges, and does a live demo of both coordinated and surprise removal on a real system to show some of the work done so far.

1. 1
© 2019 Joyent, Inc.
Joyent Technical Discussion
NVMe Hotplug
Jordan Hendricks
August 20, 2019

2. © 2019 Joyent, Inc. 2
Agenda
1. Motivation
2. Goals
3. Strategy
4. Implementation
5. Demos!
Left: “Campfire Pinecone.png” by Emeldi is licensed under Creative Commons Attribution-Share Alike 3.0 Unported
Right: “A Plug.jpg” by Maddin the brain is licensed under Creative Commons Attribution-Share Alike 3.0 Unported

3. © 2019 Joyent, Inc. 3
Motivation: Why NVMe?
● “NVM” = Non-Volatile Memory
● Standard for non-volatile storage over PCIe
● Designed for NVM instead of hard disks
○ e.g., more queues and deeper queues for I/O
● Several form factors
○ We are interested in U.2 (2.5 inch HDD form factor)
● illumos already has driver support

4. © 2019 Joyent, Inc. 4
Motivation: Why hotplug?
● “Hotplug” = generally, insertion or removal of devices on a
live system
○ Can be “coordinated” or “surprise”
■ Coordinated = notify OS first
■ Surprise = just pull/insert device
● Operational gains: no need to power off a system first
○ Guards against bad panics or errors if wrong device is
pulled

5. © 2019 Joyent, Inc. 5
NVMe SSD in slot

6. © 2019 Joyent, Inc. 6
Project Goals
1. Support coordinated insertion/removal of NVMe SSDs
2. Support surprise removal of NVMe SSDs
…for drives not in a zpool
...for drives in a zpool
...without ongoing I/O
...with ongoing I/O

7. © 2019 Joyent, Inc. 7
Project Strategy
● Hotplug support exists in the form of the hotplug
framework, integrated 10 years ago
● To find issues, tried various forms of hotplug with
NVMe drive

8. © 2019 Joyent, Inc. 8
Project Strategy
● Coordinated removal and insertion mostly worked from the
start*:
○ Removal
cfgadm -c unconfigure <slot> // offline device
cfgadm -c disconnect <slot> // power off slot
○ Insertion
cfgadm -c connect <slot> // power on slot
cfgadm -c configure <slot> // online device
*Caveat: except for OS-7494 , pci configurator issue

9. © 2019 Joyent, Inc. 9
Project Strategy
● These surprise removal experiments didn’t:
○ removing a device and plugging it back in: failed to detach
on pull; confusingly detached when plugged back in
○ pull device with ongoing I/O from dd(1M): dd hung
waiting for I/O in the kernel
○ pull a device from a mirrored zpool with no I/O: zpool
status hung waiting on I/O in the kernel

10. © 2019 Joyent, Inc. 10
Implementation: Hotplug Framework
EMPTY
POWERED
ENABLED
PRESENT
No device in slot
Device present in slot
Slot powered on
Device ready for use
Note that the slot state could jump straight from enabled to empty, but this is the general order of operations.

11. © 2019 Joyent, Inc. 11
Implementation: Hotplug Framework
● Removal case: no detach after pull
○ PCIe bridge receives a PCIe hotplug interrupt indicating the
slot status has changed
○ Hotplug code would request a state change for slot to
EMPTY from interrupt path
○ State change handler fetches state again from hardware,
see that the connection is already EMPTY, and do nothing

12. © 2019 Joyent, Inc. 12
Implementation: Hotplug Framework
● Insertion case: detach after inserted
○ PCIe bridge receives a PCIe hotplug interrupt indicating the slot
status has changed
○ Hotplug code would request a state change for slot to
PRESENT from interrupt path
○ State change handler fetches state again from hardware, see
that the connection state is PRESENT, and assumes it is
already ENABLED
○ Then, we the state change request happens, it thinks it’s going
backward (ENABLED -> PRESENT), and detaches drivers

13. © 2019 Joyent, Inc. 13
Implementation: Hotplug Framework
● Solution: Change state change transition checks in hotplug
framework to be aware of hot removal
● Internal state of hotplug framework can be out of sync if state
changes happen through surprise removal
● When in doubt, clean up structures

14. © 2019 Joyent, Inc. 14
Implementation: I/O Hang
● Both dd and zpool status hung on I/O in the kernel after hot
removal
● Want a way to notify the NVMe driver that its device is gone
● Solution?
○ Implement removal event callbacks using Nexus Driver
Interface Events
○ Add callback to the NVMe driver to fire on removal events
○ Plumb up support in nexus driver to fire removal events

15. © 2019 Joyent, Inc. 15
Implementation: I/O Hang
rootnex
pcieb pciehpc
nvme
blkdev
npe

16. © 2019 Joyent, Inc. 16
Implementation: I/O Hang
rootnex
pcieb pciehpc
nvme
blkdev
npe
npe: Implement bus ops for NDI
events.
(Events are passed up the tree
until a nexus can handle them.)
nvme: Register for remove events
on attach(9E). When remove
event fires, fail all outstanding
commands.

17. © 2019 Joyent, Inc. 17
Demo: Coordinated Removal
1: List all connections on the system.
# cfgadm
2: Meanwhile, trace whether the driver is detached…
# dtrace -n ‘fbt::nvme_detach:entry’
3: And see how many nvme driver instances we have to start.
# mdb -k
> ::prtconf -d nvme // 4 instances

18. © 2019 Joyent, Inc. 18
Demo: Coordinated Removal
4: Power off connection “Slot12”.
# cfgadm -c disconnect Slot12
5: Confirm with cfgadm.
# cfgadm
6: Confirm we see a detach dtrace output.
7. Confirm number of instances in mdb.
# mdb -k
> ::prtconf -d nvme // only 3 instances

19. © 2019 Joyent, Inc. 19
Demo: Uncoordinated Removal
1: List all connections on the system.
# cfgadm
2: Meanwhile, trace whether the driver is detached…
# dtrace -n ‘fbt::nvme_detach:entry’
3: And see how many nvme driver instances we have to start.
# mdb -k
> ::prtconf -d nvme // 4 instances

20. © 2019 Joyent, Inc. 20
Demo: Uncoordinated Removal
4: Hot removal of drive.
5: Observe dtrace output.
6: Observe nvme instances in mdb.
7: Observe in cfgadm.

21. © 2019 Joyent, Inc. 21
Demo: Uncoordinated Removal with I/O
1: Start dd command on disk.
# dd if=/dev/urandom of=/dev/rdsk/c3t1d0p0 bs=1M count=10240
2: Pull disk.
3. Observe EIO output.

22. © 2019 Joyent, Inc. 22
Demo: Uncoordinated Removal from zpool
1: Create mirrored zpool.
# zpool create test mirror c3t1d0 c4t1d0
2: Check pool status.
# zpool status test
3: Create some files.
# cd /test; echo foo > foo; cat foo;
4: Pull disk!

23. © 2019 Joyent, Inc. 23
Demo: Uncoordinated Removal from zpool
5: Check pool status.
# zpool status test
6: Read and write from the pool.
# cd /test; echo bar > bar; cat foo; cat bar;
7: Online device.
# cfgadm -c configure Slot12
8: Check pool status.
# zpool status test

24. © 2019 Joyent, Inc. 24
Remaining Work
● Hot removal from zpool with ongoing I/O panics due to OS-2743
● nvme driver sometimes fails to detach after dd exits because
blkdev still has references in devices tree (OS-7956)
● diskinfo sometimes does not list NVMe drives (OS-7940)
● Auto-online device on surprise insertion
● More work to improve NVMe operationally!

25. © 2019 Joyent, Inc. 25
Questions?