Check out our latest NVMe webinar outlining the latest test requirements for all NVMe™ and NVMe-oF™ products including the availability of NVMe-oF for TCP transports, and IOL INTERACT test tool updates for TCP testing. All great information to help get your products on the NVMe Integrators List.

1. NVMe™ Plugfest #11 Webinar
Presented by: Tim Sheehan & Kerry Munson
www.iol.unh.edu
© 2019 University of New Hampshire InterOperability Laboratory

2. 2
The UNH InterOperability Laboratory
The InterOperability Laboratory (UNH-IOL)
was founded in 1988. Main UNH campus is
located in Durham, New Hampshire, USA
The UNH-IOL is a non-profit neutral, third-
party laboratory dedicated to testing data
networking technologies through industry
collaboration.
Since 2012, NVMe Org has collaborated
with UNH-IOL to manage the Integrator’s
List based Test Program on behalf of NVMe
Organization

3. • Held at the UNH-IOL in Durham, NH
–NVMe-oF June 17-20, 2019
–NVMe-PCIe June 24-27, 2019
• Testing will include conformance and interoperability #11 test plans
• Primary purpose is to achieve NVMe.orgs Integrator’s List status for
participating company products
• Other
–NVMe/TCP large build testing - participation dependent
–DCB large build testing - participation dependent
–FC large build testing - participation dependent
NVMe Plugfest #11
3

4. ● Integrators List
● Test Plan #11 updates
● UNH-IOL NVMe Conformance tools (PCIe, RoCE, TCP, FC)
● NVMe-oF Interop Testing (RoCE, TCP, FC)
● NVMe/PCIe Interop Testing
● Future Work
● Logistics
● Q&A
Agenda
4

5. Products Listed
NVMe
Integrators List
Products
NVMe-MI
Integrators List
Products
NVMe-oF
Integrators List
Products
8 50178
Products Listed on IL as of May 7, 2019
5

6. NVMe Integrator’s List will accept:
• NVMe/PCIe drives, host platforms and switches
NVMe-MI Integrator’s List will accept:
• NVMe-MI drives
NVMe-oF Integrator’s List will accept:
• NVMe/RoCE initiators, targets, and switches
• NVMe/TCP initiators, targets, and switches
• NVMe/FC initiators, targets, and switches
NVMe Integrator’s List Products
6

7. Requirements for NVMe/PCIe SSDs:
1. Pass NVMe Conformance Tests with PC and
LeCroy Edition of the v11.0 IOL INTERACT test
tool
2. Pass NVMe/PCIe Interop Tests with 5/6 NVMe
Host Interop Platforms
Requirements for NVMe/PCIe Host Platforms and PCIe
Switches:
1. Pass NVMe Interop Tests with 5 NVMe SSDs
Requirements for NVMe-MI SSDs:
1. Pass NVMe-MI Conformance Tests with Teledyne
LeCroy test tool
NVMe Integrator’s List Requirements
7

8. Requirements for Initiators/Hosts:
1. Pass NVMe-oF Interop Tests with 2 NVMe-oF Subsystem
Interop Partners
Requirements for Targets/NVMe-oF Subsystems:
1. Pass NVMe-oF Interop Tests with 2 NVMe-oF Host Interop
Partners
2. Pass NVMe Protocol Conformance tests
3. Pass NVMe-oF Protocol Conformance tests
Two Partners could be same HW with different OS/Driver.
NVMe-oF Integrator’s List
Requirements
8

9. ● Integrator’s List
● Test Plan #11 updates
● UNH-IOL NVMe Conformance tools
● NVMe-oF Interop Testing (RoCE, FC, TCP)
● NVMe/PCIe Interop Testing
● Future Work
● Logistics
● Q&A
Agenda
9

10. NVMe
Interop
Test Cases
Tests Available
NVMe
Conformance
Test Cases
NVMe-MI
Conformance
Test Cases
NVMe-oF
Conformance
Test Cases
237 53 99
NVMe-oF
Interop
Test Cases
5
Test Plan 11 Tests Available
10

11. Conformance ToolsNVMe/ NVMe-MI/ NVMe-oF Test Plans
NVMe Interop v11.0
NVMe-MI Specification Conformance v11.0
NVMe Base Specification Conformance v11.0
NVMe-oF Interop v11.0
NVMe-oF Specification Conformance
NVMe/PCIe SSDs and Hosts
and NVMe-MI SSDs
NVMe-oF Subsystems and Hosts
(ROCE, TCP, FC)
11

12. • Over 120 test plan changes (listed in Test plan #11)
• 80 updates to base spec tests, most are refinements
• 17 new FYI tests (5 subcategories)
o Checking new Identify CNS values (Test 1.1
Cases 5-12)
o Attempt to change unchangeable feature (1.2
Case 6)
o Log Pages for Telemetry and Data Units Written
(Test 1.3 Cases 16, 17)
o Deallocate Multiple Ranges Test 2.2 Case 6
o Directives Send/Receive (Tests 1.15 Cases 1,2
and 1.16 Cases 1-3)
• 20+ Test case status changes
Conformance ToolsNVMe Test Plan #11 updates
12

13. New Tests for NVMe Base Spec Conformance for NVMe/PCIe SSDs (FYI):
o Checking new Identify CNS values (Test 1.1 Cases 5-12)
o Attempt to change unchangeable feature (1.2 Case 6)
o Log Pages for Telemetry and Data Units Written (Test 1.3 Cases 16 & 17)
o Deallocate Multiple Ranges Test 2.2 Case 6
o Directives Send/Receive (Tests 1.15 Cases 1,2 and 1.16 Cases 1-3)
Status changes for NVMe Base Spec Conformance Tests for NVMe/PCIe SSDs:
o Test 6.1 (IP → FYI) Controller Level Reset, Conventional Level Reset
o Test 9.2 case 3 (IP → FYI) NameSpace Creation, Insufficient Capacity
Conformance ToolsTest Plan #11 NVMe/PCIe Base Spec Conformance
Modifications
13

14. 1 New Test and 1 updated Test for NVMe/PCIe Interop for NVMe/PCIe SSDs:
● Hot Plug test case broken into two tests
○ New Test with I/O(FYI),
○ Existing Hot Plug Test without I/O, clarified reinsertion criteria
Conformance ToolsTest Plan #11 NVMe/PCIe Interoperability
Modifications
14

15. Conformance ToolsTest Plan #11 NVMe MI Conformance
2 Test Modifications in NVMe MI Conformance Test Plan
● Test 8.5 cases 1&2 with ECN 003 RENT changes
● Test 8.7 subtest formatting
15

16. Conformance ToolsTest Plan #11 NVMe Base Spec Conformance
Modifications for Fabrics Products
New Tests for NVMe Base Spec Conformance for NVMe-oF products:
o Checking new Identify CNS values (Test 1.1 Cases 5-12)
o Attempt to change unchangeable feature (1.2 Case 6)
o Log Pages for Telemetry and Data Units Written (Test 1.3 Case17)
o Deallocate Multiple Ranges Test 2.2 Case 6
o Directives Send/Receive (Tests 1.15 Cases 1,2 and 1.16 Cases 1-3)
Status changes for NVMe Base Spec Conformance Tests for NVMe-oF Products:
o 22 test cases made mandatory for fabrics
o 2 test cases made FYI from IP
16

17. Conformance ToolsTest Plan #11 NVMe-oF Modifications
NVMe-oF Conformance: New Test Plan
● NVMe-oF Commands
● Controller Architecture
● Discovery Service
Total of 9 tests, all FYI
NVMe-oF Interop: 1 New Test and 1 new transport
● New Test: Keep Alive Timeout (FYI)
● New Transport: NVMe/TCP
17

18. ● Integrator’s List
● Test Plan #11 updates
● UNH-IOL NVMe Conformance tools
● NVMe-oF Interop Testing (RoCE, FC, TCP)
● NVMe/PCIe Interop Testing
● Future Work
● Logistics
● Q&A
Agenda
18

19. Conformance ToolsConformance Tools / Plans
19
NVMe Base Specification
Conformance v11.0
(unique low-level PCIe test cases)
UNH-IOL INTERACT PC Edition v11.0...
… for NVMe/PCIe SSDs
UNH-IOL INTERACT LeCroy Edition v11.0 ...
NVMe Base Specification
Conformance v11.0
...Performs this required test plan...
...Performs this required test plan...
Download and
run tools today!

20. Conformance ToolsNVMe-MI Conformance Tools / Plans
20
Teledyne LeCroy T34 ...
NVMe-MI Specification
Conformance v11.0
… for NVMe-MI SSDs...Performs this required test plan...

21. Conformance ToolsNVMe-oF Conformance Tools / Plans
21
NVMe-oF Specification
Conformance v11.0
...for NVMe-oF Subsystems and HostsUNH-IOL INTERACT PC Edition...
NVMe Base Specification
Conformance v11.0
...Performs these required test plans...
Download and
run tool today!

22. • UNH-IOL INTERACT PC Edition Requirements
Conformance ToolsPC Edition System Requirements
Product OS OS Version Kernel Version
PC Edition PCIe 11.0a Ubuntu 16.04.00-16.04.03 4.10
PC Edition RoCE 11.0a CentOS 7 4.10
PC Edition FC 11.0a SUSE SLES 12 SP3 4.10
PC Edition TCP 11.0a CentOS 7.6.1810 4.19.0-rc6
22

23. • UNH-IOL INTERACT Teledyne LeCroy Edition Requirements (for
NVMe protocol)
– Windows 10 or later
– PCI Express Protocol Analyzer 10.10 build 7230
• Teledyne LeCroy T-34 Requirements (for NVMe-MI)
– FW:2.24 build 40
Conformance ToolsTeledyne-Lecroy System Requirements
23

24. NVMe or NVMe-MI SSD
IOL INTERACT Teledyne-LeCroy Edition™ HW/SW
Setup
for NVMe and NVMe-MI Conformance
NVMe and NVMe-MI: Conformance Tools
Windows 10 or Later PC
Teledyne-LeCroy PETracer
v10.10 Teledyne-LeCroy T-34
PCIe Analyzer Exerciser
USB 3.0
24

25. • When a support issue is reported, reporter will receive
an issue number, and estimated fix date based on next
release schedule
• Issues tracking internally using Jira and Git/BitBucket
for tracking and source control.
• The release notes will show what issue (with their
issue numbers) have been addressed in that release
• Enable user to make a connection to issue reported >
issue fixed
• Avoid “engineering release”, except in extreme cases
Bug Tracking ImprovementsSupport Issue Tracking Process
25

26. 26

27. ● Integrator’s List
● Test Plan #11 updates
● UNH-IOL NVMe Conformance tools
● NVMe-oF Interop Testing (RoCE, FC, TCP)
● NVMe/PCIe Interop Testing
● Future Work
● Logistics
● Q&A
Agenda
27

28. Pass NVMe-oF Interop Tests with
2 partners in a multi switch setup.
NVMe-oF Interop Test Suite
Document
NVMe-oF Interop Tests
NVMe-oF Interop Tests
Summary:
1. Discover Target
1. Send Traffic
1. Link Pull
1. Power Up
1. Keep Alive Timeout (FYI)
Server
RNIC/HBA
NVMe-oF
Target
DUT
Switch
Switch
28

29. UNH-IOL NVMe/RoCE™ HW/SW Setup
NVMe-oF Interop Tests (RoCE)
NVMe-oF RoCE Initiator
Mellanox
ConnectX4
100G RNIC
NVMe-oF RoCE Target
Edgecore 7712
100G Switch
Cumulus Linux
Edgecore 7712
100G Switch
Cumulus Linux
Mellanox
ConnectX4
100G RNICDUT Substituted
in at any position
29

30. UNH-IOL NVMe/TCP™ HW/SW Setup
NVMe-oF Interop Tests (TCP)
NVMe-oF TCP Initiator
Intel 82562V-2
10/100 NIC
NVMe-oF TCP Target
Edgecore 7712
100G Switch
Cumulus Linux
Edgecore 7712
100G Switch
Cumulus Linux
Intel 82579V
Gigabit
DUT Substituted
in at any position
30

31. UNH-IOL NVMe/FC™ HW/SW Setup
NVMe-oF Interop Tests (FC)
NVMe-oF FC Initiator
HPE StorFabric
SN1600E 32GFC
HBA
NVMe-oF FC Target
Brocade G620
32GFC
(HPE Version)
Brocade G620
32GFC
HPE StorFabric
SN1600E 32GFC
HBADUT Substituted
in at any position
31

32. • First part of Plugfest week is NVMe-oF testing, remaining part is for
large build testing
• Extend the week of NVMe testing to perform large build interop
testing with all participants.
• Weekly call to setup test cases and other logistics (Thursdays 2PM
EDT)
• Large builds will only occur if enough interest from participants
• Register for the NVMe-oF Plugfest
– https://www.iol.unh.edu/event/2019/06/nvme-plugfest-11
Conformance ToolsNVMe-oF Large Builds
32

33. ● Integrator’s List
● Test Plan #11 updates
● UNH-IOL NVMe Conformance tools
● NVMe-oF Interop Testing (RoCE, FC, TCP)
● NVMe/PCIe Interop Testing
● Future Work
● Logistics
● Q&A
Agenda
33

34. NVMe/PCIe Interop Systems
MAKE MODEL BIOS OPERATING SYSTEM DRIVER VERSION CPU MODEL
ASRock ASRock Z97
Extreme 6
P2.3 Boot Test Machine Inbox Microsoft NVMe
Driver/NVMe Driver 1.0
Intel i5-4690K @3.50
GHz
ASUSTeK
Computer Inc.
ASUS ROG
STRIX B350-F
0809 Windows 10 Inbox Microsoft NVMe Driver AMD Ryzen 3 1300X @
3.50 GHz
ASUSTeK
Computer Inc.
ASUS X79 Deluxe
LGA 2011
0701 Ubuntu Linux
16.04.1
Linux Kernel 4.4.3 NVMe
Driver
Intel Xeon CPU E5-2667
@ 2.9 GHz
Dell EMC Dell PowerEdge
R740xd
Windows 10
Dell, Inc. Dell PowerEdge
R720
2.2.2 Windows Server
2012 R2
Inbox Microsoft NVMe Driver Intel Xeon CPU E5-2640
@ 2.5GHz
ASUSTeK
Computer Inc.
Asus Z170-A 1902 Ubuntu 16.04.5
Kernel 4.8.0-0408
Linux Kernel NVMe Driver
Ver 1.0
Intel Core i5-6500 @
3.20GHz
GIGA-BYTE
Technology
GIGA-BYTE
H170-Gaming 3
F22e Ubuntu 16.04.1 Linux Kernel 4.4 NVMe
Driver 1.0
Intel Core i3-6100 @ 3.7
GHz
ASUSTeK
Computer Inc.
Asus Z9PA-U8 5507 Boot Test Machine Inbox Microsoft NVMe
Driver/NVMe Driver 1.0
Intel Xeon ES-2667 @
2.9GHz
https://www.iol.unh.edu/testing/storage/nvme/equipment 34

35. ● Integrator’s List
● Test Plan #11 updates
● UNH-IOL NVMe Conformance tools
● NVMe-oF Interop Testing (RoCE, FC, TCP)
● NVMe/PCIe Interop Testing
● Future Work
● Logistics
● Q&A
Agenda
35

36. • Goal: Continually Evolve the Test Program in order to Keep up with NVMe
Specifications
• Roadmap
• 2H19’ focused on TP2008, expected to be ratified around time of Plugfest #11
▪ TP2008 describes updates for Test Plan #12 for NVMe, NVMe-MI, NVMe-oF
Conformance and Interop, to be performed at Plugfest #12 in Fall 2019.
▪ Addresses more ECNs and TPs to NVMe v1.3
• Late 2019, work starts on next TP for Plugfest #13 in Spring 2020.
Goals + MindsetFuture Work
36

37. ICC Calls
Interop and Compliance Committee (ICC)
Chair: Ryan Holmqvist, Microsemi
Meets every other Tuesday 2PM Eastern / 11AM Pacific
Announcements/Meeting invites made on ICC reflector
ICC Calls
37

38. • Integrator’s List
• Test Plan #11 updates
• UNH-IOL NVMe Conformance tools
• NVMe-oF Interop Testing (RoCE, FC, TCP)
• NVMe/PCIe Interop Testing
• Future Work
• Logistics
• Q&A
Agenda
38

39. Download the latest tools and run them!
● Tools used at the plugfest are available now
● Use them to pre-qualify your products
● Link to licensing agreement
–https://www.iol.unh.edu/solutions/test-tools/interact
● Link to UNH Box PC Edition IOL INTERACT Software
–https://unh.box.com/s/cktgos25cjfk6alclromuvqfn1lgebgd
● Link to UNH Box LeCroy Edition IOL INTERACT Software
–https://unh.app.box.com/folder/75601888099
Be Prepared!
39

40. Bring 2+ Samples of each product
• Conformance tests will only require 1 sample
• Interop tests require that 2 samples be used
simultaneously
• Bring any necessary PCs, tools, cables that will be necessary to re-program
or re-flash your device
• If you have a larger product (e.g. large PCIe dev board), bring your own PCIe
flex or riser cables
• If you have a non CEM form factor, please bring it, and also please bring at
least 2 of your own adapters - this will facilitate interop testing
• Adapters available from serialcables.com and teledynelecroy.com
Be Prepared - PCIe!
40

41. • Bring host system for products to run in
• Bring required cabling for your product
(including break out cables)
• Bring any necessary PCs, tools, cables
that will be necessary to re-program or
re-flash your device
• Short Range, multimode transceivers
Be Prepared - Fabrics!
41

42. Available!
● We have a soldering station in the Plugfest Room.
● Wired and wireless internet provided
● Power cords, mice, keyboards, monitors
42

43. No Photos
● Please respect confidentiality of your colleagues.
● All companies have agreed to UNH-IOL Usage
Agreement
● UNH-IOL may take some ‘generic’ photos.
● If you would like a photo of your team, or your own
equipment setup, please check with UNH-IOL
staff first.
43

44. Travel
Visa Invitation Letter Contact:
kerry.munson@iol.unh.edu
Shipping Address:
UNH-IOL NVMe Plugfest
Attn: Kerry Munson
21 Madbury Rd Suite 100
Durham, NH 03824
📞: 603-862-0090
All equipment must arrive by June 14th for
Fabrics Plugfest and June 21st for PCIe
Plugfest.
Airports:
MHT - Manchester Boston Regional Airport
( 🕐1 hr)
BOS - Boston Logan International Airport
( 🕒1hr 30mins)
Hotels, Travel, Parking Info:
https://www.iol.unh.edu/about/visit
Train: 🚄
Amtrak - Downeaster
Boston North Station > Durham-UNH
http://www.amtrakdowneaster.com/
44

45. Parking
Parking code will be provided
to registrants
UNH Campus Map:
https://www.iol.unh.edu/sites/
default/files/images/direction
s/iol-campus-map.pdf
45

46. Logistics
● Plugfest runs Monday - Thursday
● Doors open at 8AM, testing until 6PM
● Kickoff meeting Monday @ 11AM
● Light breakfast and lunch provided each day
● Device schedule will be made after registration closes
46

47. Registration
Register here:
NVMe-oF Plugfest - https://www.iol.unh.edu/event/2019/06/nvme-plugfest-11
NVMe PCIe Plugfest - https://www.iol.unh.edu/event/2019/06/nvme-pcie-plugfest-11
Registration will close May 17, 2019 (Fabrics), May 24, 2019 (PCIe)
Registration Limits:
● At registration 2 unique products may be registered.
● All other products will be waitlisted, and accepted on a space-available basis.
● At registration you can register 4 engineers.
● Additional engineers will incur an additional fee.
47

48. Contact Information
48
Kerry Munson
Operations Manager,
Datacenter Technologies
Kerry.Munson@iol.unh.edu
Tim Sheehan
Manager, Datacenter
Technologies
Tim.Sheehan@iol.unh.edu

49. Questions?
49
Please type any questions you have into the
“questions” window
Twitter @UNH_IOL | Facebook @UNHIOL
LinkedIn @UNH InterOperability Lab