RDMA programming design and case studies – for better performance distributed applications
•Download as PPTX, PDF•
1 like•599 views
Open Source Summit, 2018
![2Copyright©2018 NTT corp. All Rights Reserved.
NTT Confidential
• Remote Direct Memory Access(RDMA) is low latency and low cpu
overhead, but hard to implement
• In simple micro-benchmark, RDMA’s latency is about one fifth of
IPoIB’s latency (IPoIB : IP over Infiniband)
RDMA features, potential
1
10
100
1000
10000
100000
1000000
8 32 128 512 2K 8K 32K 128K 512K 2M 8M 32M 128M 512M
Latency[us]
Data size[byte]
IPoIB
RDMA
3usec vs 15usec](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
What's hot
What's hot (20)
Similar to RDMA programming design and case studies – for better performance distributed applications
Similar to RDMA programming design and case studies – for better performance distributed applications (20)
More from NTT Software Innovation Center
More from NTT Software Innovation Center (20)
Recently uploaded
Recently uploaded (20)
RDMA programming design and case studies – for better performance distributed applications
- 1. Copyright©2018 NTT corp. All Rights Reserved. RDMA programming design and case studies – for better performance distributed applications NTT Software Innovation Center Yoshiro Yamabe
- 2. 2Copyright©2018 NTT corp. All Rights Reserved. NTT Confidential • Remote Direct Memory Access(RDMA) is low latency and low cpu overhead, but hard to implement • In simple micro-benchmark, RDMA’s latency is about one fifth of IPoIB’s latency (IPoIB : IP over Infiniband) RDMA features, potential 1 10 100 1000 10000 100000 1000000 8 32 128 512 2K 8K 32K 128K 512K 2M 8M 32M 128M 512M Latency[us] Data size[byte] IPoIB RDMA 3usec vs 15usec
- 3. 3Copyright©2018 NTT corp. All Rights Reserved. NTT Confidential • To make use of RDMA potential, I tried to apply RDMA to MXNet, a distributed deep learning framework – MXNet adopts Parameter Server mechanism for distributed learning • RDMA was seemed to be effective for this model – There are two times of communication between worker and server for each batch About case studies Parameter Server worker 2. Push parameter to Server node 4. Pull parameter from server node 1. Calculate Parameter on each worker node 3. Aggregate parameters from all worker nodes worker worker
- 4. 4Copyright©2018 NTT corp. All Rights Reserved. NTT Confidential Data flow comparison – existing implementation and RDMA calc aggr worker server calc push pull Parameter Server worker 2. push 4. pull1. calculation 3. aggregation calc aggr calc pullpush worker server Existing(ZeroM Q) RDMA(ibverb s) pack/unpack message copy user->kernel copy send process(short) recv process • By applying RDMA, I can reduce 2 memcpies per communication, total 4 copies per push and pull Reduced overhead
- 5. 5Copyright©2018 NTT corp. All Rights Reserved. NTT Confidential • RDMA mechanisms – RDMA operations(Operations : WRITE(put), READ(get), etc…) – RDMA operations’ options(inline send, immediate data, etc…) – Way to detect communication completions(Polling or MSI-X interrupt) • Kernel layer mechanisms – ring buffer for asynchronous communication (≒ socket buffer) – Separating detect completion thread RDMA implementation Tunings Types Tunings STEP1 STEP2 RDMA mechanisms RDMA operations 〇(using RDMA WRITE) 〇(=STEP1) RDMA operations’ options 〇 〇(=STEP1) Way to detect completion - 〇 Kernel layer mechanisms Ring buffer 〇 〇 Separating detect completion thread - 〇
- 6. 6Copyright©2018 NTT corp. All Rights Reserved. NTT Confidential Result(Step1 vs Step2) 0 10 20 30 40 50 60 70 1 2 3 Timecost[s] # of worker STEP1 STEP2 • Step2 implementation is 1.5x faster than Step1 implementation – “step0 implementation” which doesn’t have ring buffer is many times slower than step1 implementation 1.5x speed up
- 7. 7Copyright©2018 NTT corp. All Rights Reserved. NTT Confidential Result(But…) 0 10 20 30 40 50 60 70 1 2 3 Timecost[s] # of worker zeromq + IPoIB • Existing implementation(zeromq) is fast!! – Step2 implementation is faster than it, but a bit… （There are still other tuning so RDMA can be faster！！） STEP1 STEP2
- 8. 8Copyright©2018 NTT corp. All Rights Reserved. NTT Confidential • RDMA is efficient for reducing latency, cpu overhead, but there are many techniques to achieve high performance – Selecting optimal options, flags, and redesign functions! – Porting a part of kernel layer tunings into user layer programs • As a matter of course, it is need to choose appropriate applications for RDMA – CPU centric, network bottleneck, etc… – In this workload and environment, network load may be not adequate for confirming RDMA effect with lightweight change • My Conclusion – To enjoy RDMA effect easily, libraries providing RDMA design patterns are needed – “true zero-copy RDMA implementation” is needed for higher performance Conclusion
- 9. 9Copyright©2018 NTT corp. All Rights Reserved. NTT Confidential User memory • RDMA can reduce overhead and accelerate applications by these two features – zero-copy • NIC can read/write data to user memory directly – cpu-bypass • When using one-sided RDMA, client can send data without involving server’s cpu The Features of RDMA User memory kernel memory memcpy NIC kernel memoryNIC memcpy User memory User memory kernel memory RDM ANIC kernel memory RDM ANIC IP RDMA
- 10. 10Copyright©2018 NTT corp. All Rights Reserved. NTT Confidential • Various resources for RDMA – Memory Region(MR) • RDMA NIC can only read data on MR / write data to MR – Completion Queue(CQ) • Communication’s success/failure information(completion) are pushed into CQ, and program can get it by pop from CQ by two ways – Polling or completion channel(MSI-X interrupt) – Queue Pair(QP) • similar to socket • By posting request to QP, RDMA NIC execute data transfer referring to it. – RDMA Operations • SEND, RDMA WRITE, RDMA READ, ATOMIC – SEND needs for receiver side to post receive request to QP(two-sided RDMA) – Others don’t need for receiver side to any operation (one-sided RDMA, cpu-bypass) • Service Types – RC, UC, UD, RD – RC can detect packet loss and re-send, but slower than other UC and UD RDMA Programming Features
- 11. 11Copyright©2018 NTT corp. All Rights Reserved. NTT Confidential • Ring Memory Region – By using ring MR, sender can issue RDMA independent of receiver status – It’s similar to IP’s socket buffer in kernel space • Create additional thread for getting completion from CQ – Getting completion in send function in application is inefficient because IP’s send function returns when send data pass to kernel – Using polling to get work completion because CPU resource waste can be avoided by pinning thread to core Various techniques of RDMA programming for this attempt ・・・ A B C D E RING MR CQ polling Polling Thread RDMA NIC generate completion write data process data Receiving Thread pass informatio n receiver side RDMA Operation
- 12. 12Copyright©2018 NTT corp. All Rights Reserved. NTT Confidential Experimental environment OS Ubuntu16.04 CPU Intel® Xeon® CPU E5-2697 v3 @ 2.60GHz # of core, node 2 numa nodes and 14core 28thread for each numa node Memory 256GB Network Mellanox ConnectX-5, 100Gbps cable GPU Tesla V100 • Using 1 server nodes and 1-3 worker nodes • GPU and HCA are on the same numa node(e.g. node1) on all machines, and using numactl to avoid “remote access” • Using Cifar-10 and ResNet50 to training • Environment
Editor's Notes
- Hello, I’m Yoshiro Yamabe, I’m working NTT Software Innovation Center. Now I tackle about Remote Direct Memory Access, RDMA technologies. Today, I’ll talk about my experience, as “RDMA programming design and case studies, for better performance distributed applications”.
- At first, I talk about RDMA protocol’s merit, demerit and potential. RDMA provides low latency and low cpu overhead, because of two major features, zero-copy and cpu-bypass. So the potential of RDMA is very attractive, for example, in simple ping-pong program, RDMA’s latency is one fifth of IPoIB’s latency. However, it is difficult to implement than existing TCP/IP programming. For example, to establish RDMA connection, it is need to exchange various information by using IP. And there are many other complications, so this benchmark’s IPoIB program is about three hundred lines with comments and spaces, but RDMA program is about 800 lines.
- I thought distributed deep learning is compatible for RDMA, so I chose MXNet as a distributed deep learning framework and tried to apply RDMA to it. MXNet adopts parameter server mechanisms for distributed learning. Parameter server consists of two types of nodes, one is worker node which computes parameter, and the other is server node which gathers parameter from worker nodes, and aggregates them, and finally, scatter parameter to all worker nodes. So the processing flow of each batch is, at first, calculate parameter on workers, second, each worker push parameter to server, third, server aggregates them, last, server push parameters to each workers. Then next batch will start.
- Zoom up this flow, the detail is like this. In existing implementation, zeromq, there are user to kernel or kernel to user memcpy for each send/recv. If RDMA is applied, these copies are reduced instead of increasing additional send/recv processes. But memory copy latency is large, so RDMA can reduces overhead. Red box in this figure indicate memcpy from data update memory area to communication memory area, and ideally, this types of copies should be reduced, But it is out of scope in this implementation.