The document discusses using RDMA over Converged Ethernet (RoCE) for high performance data movement between KISTI and LBL. It describes how RDMA allows direct data placement through one-sided operations like RDMA write and read, avoiding CPU overhead. It also discusses challenges in using RDMA over wide area networks and for bulk data transfers, and experiments using GridFTP and a prototype FTP-like transfer application over RDMA.

1. RDMAoE collaboration with KISTI
Tuesday 6/7/2011
10:00am-11:00am (50B-2222)
mbalman@lbl.gov

2. RDMA for High Performance Data
Movement

Network I/O operations are costly:
− CPU load
− Context switching
− Memory latency

Zero-copy networking
− NIC copies data directly to/from application
memory

IB transport (HPC applications)

iWARP (TCP stack / TOE)

3. RDMA model

One sided operations

Get/Put semantics

Send/receive

Direct data placement

RDMA Write

RDMA Read

Asyschronous
− Work Queue (send queue – receive queue)
− Completion Queue

4. RDMA Programming Model

Objects

Queue Pairs (protection domain)

Send queue (RDMA write, RDMA read)

Receive queue

Modify state

Completetion queue (poll)

Memory region (MR)

Functions (verbs)
− IB (libmlx4) iWARP (libcxgb3)

Librdmacm (connection setup)

5. RDMA/iWARP

Implicit RDMA support

Explicit RDMA support

iWARP
− encapsulate RDMA traffic at a high level
− Use TCP stack
− Without TOE is it beneficial?

6. Alternative Approaches

RDMA over Converged Ethernet (RoCE)
− Lightweight RDMA transport over Ethernet

Widely deployed technology

Support kernel bypass

OFED 1.5.1 supports RoCE

SoftRDMAs...
− SoftRoCE (OFED 1.5.1 supports softRoCE)
− SoftiWARP (new TPC kernel stack)

7. Hidden Cost

Memory Registration
− RDMA Read/Write

Connection Setup
− Librdmacm
→ Bulk data movement?

Asynchronous Model
− Buffer Management

8. Challanges in Bulk Transfer

Application Level Adjustments

Request Aggregation
− Small data files
− Does FTP like transfer mechanism is appropriate
for RDMA?

File System Overhead
− Asynchronous Operations

Connection Caching / Multiple Connection?

9. Local Area / Wide Area

IB RDMA designed for local area
− How does RDMA perform in Wide Area?

iWARP
− No promising results - Over TCP (with TOE?)
− SoftiWARP ???

RoCE
− Isolated traffic ? / much less CPU usage
− softRoCE?

10. GridFTP over RDMA

XIO driver for GridFTP
− Experimented using Chelsio cards (cxgb3)
− 10GE
− WAN testing in progress!
− Local area: 910MBbps – 1175MBps
− Much better than GridFTP over TCP

Much less CPU load (1/2)

11. FTP100 – FTP over RDMA

Experimented with Mellonox Cards
− Local area – 10GE
− iWARP

Did not perform well compared to TCP
− No significant gain
− RoCE tests

In progress (have some initial results)

Limited by the disk performance

Mem2mem:
− Can already saturate the 10GE link

12. What is Next?
Experiments RDMA model over WAN

SoftiWARP from IBM Zurich
− TCP kernel stack implementing/defining RDMA
iverbs

SoftRoCE – OFED 1.5.2-rxe distribution
− Multiple connections?

13. Transfer Applications over RDMA

Simple Client/Server:
− Developing a prototype for transferring climate
dataset using RDMA protocols
− Asysnchronous memory management module

Application level tuning?
− Memory regions (max/min?)
− Multiple QPs

14. Climate Analysis
Climate Applications are Data-Intensive

Shared data repository:
− Data files needs to be downloaded for further
processing and analysis
− Data retrieval is the main bottleneck
− Multiple clients (working as VM instances)

Can not depent on HW support

SoftRoCE ? softiWARP

15. What can we do for WAN testing?

Q&A?
→ https://sdm.lbl.gov/climate100/