The Infiniband Trade Association has released the RoCEv2 specification to enhance the capabilities of RoCE for evolving enterprise data centers transitioning to hyperscale environments. This update provides low-latency and efficient data transfer, leveraging RDMA to minimize CPU involvement and improve network performance. The RoCEv2 specification includes new functionalities such as Layer 3 routing via UDP headers while remaining transparent to applications and network infrastructures.

1. InfiniBand Trade Association
NEWS UNDER EMBARGO UNTIL
TUESDAY, SEPTEMBER 16 at
6:00AM PACIFIC TIME
RoCEv2 Specification Release

2. •Extends capabilities of original RoCE specification announced in 2010
•Addresses needs of evolving enterprise data centers moving to hyperscale networking environments
IBTA Announces RoCEv2 Specification
© InfiniBand Trade Association
2

3. Global member organization dedicated to developing, maintaining and furthering the InfiniBand specification
•Architecture definition
–RDMA software architecture
–InfiniBand, up to 56Gb/s and 168Gb/s per port
–RDMA over Converged Ethernet (RoCE)
•Compliance and interoperability testing of commercial products
•Markets and promotes InfiniBand/RoCE
–Online, marketing and public relations engagements
–IBTA-sponsored technical events and resources
InfiniBand Trade Association (IBTA)
© InfiniBand Trade Association
3

4. •Remote Direct Memory Access
•The movement of data between servers’ user space without CPU involvement
–Traditional data movement utilized TCP/IP, many copies, and significant CPU overhead
–RDMA utilizes hardware offloads to move data faster with less overhead
What is RDMA?
4
© InfiniBand Trade Association

5. Without Remote Direct Memory Access
5
© InfiniBand Trade Association
Server - Initiator
Sockets
NIC Driver
Buffer
Transport Protocol Driver
Buffer
NIC Driver
Buffer
Transport Protocol Driver
Sockets
Application
Server - Target
NIC
Buffer
NIC
Buffer
Application
Buffer
Buffer
Buffer
Buffer
Buffer

6. With Remote Direct Memory Access
6
© InfiniBand Trade Association
Server - Initiator
Sockets
NIC Driver
Buffer
Transport Protocol Driver
Buffer
NIC Driver
Buffer
Transport Protocol Driver
Sockets
Application
Server - Target
RNIC
Buffer
RNIC
Buffer
Application
Buffer
Buffer
Buffer
Buffer
Buffer

7. Essential for evolving enterprise data center and hyperscale networking infrastructure
•I/O is central to achieving highest performance
•Efficient computing reduces power, cooling and space requirements
•OS bypass enables fastest access to remote data
•Scalable storage to meet growing demand
•Delivers direct access to data over the WAN
Benefits of RDMA
•Low latency and CPU overhead
•High network utilization
•Efficient data transfer
•Support for message passing, sockets and storage protocols
•Supported by all major operating systems
Why RDMA?
© InfiniBand Trade Association
7

8. •RDMA transport over Ethernet
–Efficient, light-weight transport, layered directly over Ethernet
–Takes advantage of PFC (Priority Flow Control) in DCB Ethernet
–IBTA standard, supported in OFED 1.5.1
•Lowest latency in the Ethernet industry
–1.3μs end-to-end RDMA latency
–Enables faster application completion, better server utilization and higher scalability
•Tremendous support momentum by ecosystem
–Cloud service providers, DB Vendors, Financial ISVs, Server & Storage OEMs
–Entire Ethernet management ecosystem is available
RoCE
8
© InfiniBand Trade Association

9. •TCP wasn’t designed for RDMA
•RoCE supports lowest latency
•Adapter power comparable to plain NICs
Most Efficient RDMA over Ethernet
© InfiniBand Trade Association
9

10. •Extends RoCE functionality & scalability
•L3 routing through the use of UDP headers
•Transparent to applications and underlying network infrastructures
–InfiniBand Architecture followed OSI model closely
–RoCEv2 only modified third layer
–Frames generated and consumed in the NIC (below the API)
•Enables standard network mechanisms for forwarding, management, monitoring, metering, accounting, firewalling, snooping, and multicast
RoCEv2 Specification Details
© InfiniBand Trade Association
10

11. Protocol Stacks, Side by Side
11
© InfiniBand Trade Association

12. Frame Format
12
© InfiniBand Trade Association

13. RDMA Use in the Data Center
© InfiniBand Trade Association
Scale up with Single Ethernet L2 Domain
(from end node perspective)
L2
L2
L2
Plain L2 or TRILL or PBB or
Virtualization Overlays
13

14. RDMA Use in the Data Center
© InfiniBand Trade Association
L2
L2
L2
14

15. RoCE Use Case – Microsoft Azure
© InfiniBand Trade Association
15
Microsoft Keynote
Albert Greenberg
SDN in Azure Infrastructure

16. RoCE Use Case – Dell Fluid Cache SAN
© InfiniBand Trade Association
16

17. RoCE Use Cases – Zadara iSER
© InfiniBand Trade Association
17

18. RoCE Use Case – Applied Micro ARM
© InfiniBand Trade Association
18

19. Thank You

20. IBTA Members
© InfiniBand Trade Association
20
3M
Advanced Photonics, Inc.
Agilent Technologies, Inc.
Amphenol Interconnect Products
Annapurna Labs (US) Inc.
Anritsu
AppliedMicro
Bay Microsystems
Broadcom
Bull SAS
Cisco
ConnPro Industries Inc.
Cray, Inc.
Emulex
FCI USA Inc.
Finisar Corporation
Foxconn
Fujikura America, Inc.
Fujitsu Limited
Hewlett-Packard
Hitachi
IBM
Intel Corporation
Lamprey Networks, Inc.
Lawrence Livermore National Laboratory (LLNL)
Lorom Cable & System Design
LUXSHARE-ICT Co., Ltd.
Mellanox Technologies, Inc.
Mercury Systems
Meritec
Microsoft
Molex Inc.
NetApp
Netronome
Oracle
QLogic
SAE Magnetics (HK) Ltd.
Samtec
Semtech
Silicon Graphics, Inc.
Software Forge, Inc.
Sumitomo Electric Industries
TE Connectivity
Tektronix
The Siemon Company
Time Interconnect Ltd.
UNH InterOperability Lab
Unisys Corporation
Volex Inc.
Woodward McCoach, Inc.
Yamaichi Electronics USA
BOLD = Steering Committee member