More Related Content Similar to Docker infiniband (20) More from Syoyo Fujita (7) Docker infiniband2. Why InfiniBand?
• Faster networking
• 3.2 GB/s on QDR
• Low laterncy
• ~3 us
• de-facto inter-connect standard in HPC
environment
Thursday, February 13, 14
5. My contribution (2/2)
•
Thursday, February 13, 14
OFED userland port for OpenIndiana(Solaris)
•
https://github.com/syoyo/oi-build/tree/
ofed-1.5.4.1/components/open-fabrics
8. Existing research
• Performance Evaluation of Container-based
Virtualization for High Performance
Computing Environments
• Xavier, M.G. et. al. PDP2013
• http://marceloneves.org/papers/pdp2013containers.pdf
Thursday, February 13, 14
15. • Privilege : on
• bind /dev/infiniband
$ sudo docker run -privileged
-v /dev/infiniband:/dev/infiniband
-i -t centos6-ib /bin/bash
Thursday, February 13, 14
17. LXC Container
# ib_read_bw 192.168.11.16
…
----------------------------------#bytes
#iterations
BW peak[MB/sec]
BW average[MB/sec]
65536
1000
3008.83
3008.80
----------------------------------------------------
MsgRate[Mpps]
0.048141
# ib_write_bw 192.168.11.16
…
----------------------------------#bytes
#iterations
BW peak[MB/sec]
BW average[MB/sec]
MsgRate[Mpps]
65536
5000
3066.07
3066.05
0.049057
--------------------------------------------------------------------------------------# ib_read_lat 192.168.11.16
…
----------------------------------#bytes #iterations
t_min[usec]
t_max[usec] t_typical[usec]
2
1000
1.66
20.13
2.65
--------------------------------------------------------------------------------------# ib_write_lat 192.168.11.16
…
----------------------------------#bytes #iterations
t_min[usec]
t_max[usec] t_typical[usec]
2
1000
0.80
7.12
0.86
---------------------------------------------------------------------------------------
Thursday, February 13, 14
18. Native
$ ib_read_bw 192.168.11.16
…
----------------------------------#bytes
#iterations
BW peak[MB/sec]
BW average[MB/sec]
MsgRate[Mpps]
65536
1000
3015.43
3011.54
0.048185
--------------------------------------------------------------------------------------$ ib_write_bw 192.168.11.16
…
----------------------------------#bytes
#iterations
BW peak[MB/sec]
BW average[MB/sec]
MsgRate[Mpps]
65536
5000
3065.79
3065.63
0.049050
-------------------------------------------------------------------------------$ ib_read_lat 192.168.11.16
…
----------------------------------#bytes #iterations
t_min[usec]
t_max[usec] t_typical[usec]
2
1000
1.68
39.48
2.67
--------------------------------------------------------------------------------------$ ib_write_lat 192.168.11.16
…
----------------------------------#bytes #iterations
t_min[usec]
t_max[usec] t_typical[usec]
2
1000
0.80
4.70
0.86
--------------------------------------------------------------------------------------Thursday, February 13, 14
20. Summary
• Docker + InfiniBand works!
• No virtualization overhead!
Thursday, February 13, 14