Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage at OpenStack Summit Vancouver

1. Copyright©2018 NTT Corp. All Rights Reserved.
:
:

2. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
) 4
424 4. ) 1 431 1 ) . (11
1. 4 3
(1 1
. 1 . 1.
, 1 1 . 4 4 1

3. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• 1 . : .
• . .
- .

4. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• . 1 / , / , ()) : ,
, 1 , , 1 . , .
, , . , , A ,
: , . , / . :,
: . : .
.

5. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• & : - :
• 1
• - - / - 5 - 5 / 5 -
- 5 - - 5 / 5
• - - - 5 5 5 5
corevo Computing Infrastructure (On-Premise)
: . -
Compute (GPU Servers) Storage
Dashboard
Monitoring
High Speed Network
AI Framework Stacks
datamodels
- -
5 5
- 5
- - -

6. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• .
1 1
AI Researchers, Framework Design
Facilities, Network Design
Scale-Out Storage Design
Monitoring/ and Operation Design
C D 6
6 2 2A2 2
2
2 2 22 2 2
2 1
2 2 6 22
6 6 2A2 2
2 2 2
2 6
2 2 6 22

7. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
. 1 .1
GPUs
Millanox 100G EDR Infini Band Backend
AI Framework Stacks
APPAPP
Bare-Metal HW Stack
Storage
7 - 7
Managements
Tesla K80 GTX1080 GTX1080
Tesla K80
7 - 7 7 7
TCP 10G Ethernet Service
TCP 1G Ethernet Management, IPMI
Software Stack
Ubuntu
-
-
Docker KVM
Guest OSGuest OSGuest OS
Dashboard Monitoring Ansible
Swift
- 7
NFS
Chainer MN Chainer
SwiftFS

8. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential

9. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• : . : A .
• 9 ,
• . A: : / :
• 9 99 9 :
• : C . 9 C ,D
• . / :/
• , 99 9 : D : 9 :
• . ?. :
• - 9 B A9
• : 9 B B
D . / : ?

10. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
.
1 : 70: 6 2-2 7 70 1 : -2//.9.6 . 9 2/2 2 26 . 20.6 . 126. . 96260 -.. . 96260 2
1 . 20 . /97 9 -2 276 126.
. 96260 7 -.. . 96260 9. 20 - 6-
7 260 9.:7 9 .:

11. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
.
7- 0 0: 7- 2 -7 :7 1 7- - 2 / 71 0 1 22
- 0- 2: - . 7 1 1 /
0 1 / /2- 2 / 7- 1 1 1 / : 7- 2- -..- 1 - - 0 2

12. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
.
• : A :A :
2 A A A
: 2
• A D : D A -
,. ::
AA A 1 A : A A 2A A A 1 A A : AD
• :: 2 A A
: D A A A A
: 2
• A 2 A A
: A A
: , :: : A , :: :
/ 2 A :: : 2 A A A : : F

13. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
A G C
C ( G - D : -
:
A D : . .
CD D
D:
K 2G D K
2G D
K
2G D
K
2G D
K
: I D OF ( OF OF
( OF
OF OF OF ( OF
. D C F R R P P R
D D )
2 /
3))1 AGIGMG
2 /
.
-
( 3
2 / 3))1 )
DGG
2 /
H F2 /
L IF K
, I
). D
2( 2(
2 /
2( 2 / 2(
2 /
2( 2 /
CD D
D
A HIG I G I E NGIC M IO A A
)A F I23 LHHGIK K H I DD D E N KA 2 / K F IDO K
GM I A A H FK I GFF K F F F F 3))1

14. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• . : . . . . .
A
• 4 14 341
• 3 1. . 4 4 14 341 34
• . : . : : : . :
- .
41 33 41 4- . -. 1 . 1-
. 3 2 1 4 -. 131 . . 2 .
1 1 . -
1414/
C . . . .

15. (Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
.
/ / / /
3 31
/ 13 / 13 / 13 / 13
/ 3 1 A 1/
3/2 3
2/ / 2 0A 3
3 13 3
31 2/ / 31 2/ / 31 2/ /
3A / 3 3A / 3 3A / 3 3A / 3
. / . / . / . /
A A / 3
5 23
5 A 5 A 5 A 5 A
) 3 /5 5 3 5
/ 32A13 /4 3
0/1 / 2 0
/
31 2/ /
. /

16. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• . :
.
• -M> M.66
• 7I 2 991 C KC> .H D H L C
• 6--4
• 46: 70/.
• .
9L L EE H >H D H L C K
3C D L L C C K CIL L F KL H>
• KL H> , NH D H> K
9LHIK EE H H L C K N L C C C CK >

17. *Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
. . - .
(
)
*
,
( ) * ,
5 = D G 9 G D
6 D 47 D9 G DI 8 G D
4D=G
23056
G 8 1
D G

18. ,Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
- .
(
)
*
( ) *
78A8DI8
0A8EG 5 B G
PN D
3D D 8 D 8 8 D
.8 B 8A D
I8
2B8 3
* GA8 , ( DG G (146G

19. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• . . . . . . .
: . : :
• 1 /1/ 1 /1
• . . : : . . . :
.
• . . ./ : : -
• : : : . :/ /
: . . : : A
D . .

20. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• 3: 3 .3 . 3
• 2 0
•
• 0 0
• , . 3: 3 .3 . 3
• 2 0
• 0
• ,
. . .3 . 3 3.

21. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• - : D . ::
3 3 )
( 3C 3 3 C D C
• ( . : C3 C
3 C .
) . : 3
D 3 . :

22. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
. . .
IB SW IB SW
IB SW IB SW IB SW IB SW
2
-
2
-
CPU CPU
()
HCA
()
HCA
GPU GPU GPU GPU GPU GPU
-
-

23. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• . .
• -366078 8 1 2 ) (
3 . - . .

24. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• --
• / 2 2 / B / 1 B /44 1 1/
• B: 4 2 9 8 B/9 B: 4
B 9 8 9 18 8
- 3 3 3. 3 3 -
( 9/
/1
/ -.
IB SW IB SW
IB SW IB SW IB SW IB SW
9/ /9
) 19B 2 4/

25. )Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
3 : 3 ? ?3 3 3 3
• 3 3 3 3 3 3
• 4ECD 37: AH EM - 6 H 7 5A GIM A A A
• 3 3 3 :
• 1 G 9I AM 5HMAG AIH 7: 2) ,
• 3AHA MEIH G HA ) ) 3/ 7 5A Q ( GIM 7:
• 75 , 3/ IHHA MEIH AMPAAH 7:
• 3 : : : 3 3 3
C :A . 3 3
C :A 3 3
CPU CPU
75
7 5A
9
7 5A
9
GPU GPU GPU
GPU GPU GPU
IHCA IHCA
54 .- 5HBEHE H 4I M D HHAG . MI

26. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• 1 3 1 3
D 1 3 . 1 1 ::A 1
C A
CPU CPU
IHCA IHCA
GPU GPU GPU GPU GPU GPU

27. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• 3: ) : )
• 2AA ( 2 2B2 B 2 A7 G
• ( A CAB B B A2 B D
• 3: )
• 2AA ( 2 2B2 B 2 A7 G 2B +C B A
• + 2 D ) ) AB 2 2 B ( C
( : . ( 3
GPU GPU
GPU
IHCA
GPU
IHCA
CPU
CPU CPU

28. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• 1 3 1 3
D 1 3 . 1 1 ::A 1
C A
CPU CPU
IHCA IHCA
GPU GPU GPU GPU GPU GPU

29. ,Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
. 3
(
)
*
,
( ) * ,
7 D G= KI K = G=
8 EGI 059 G K D GM I = E DK G=
5IGKE
34178
K D 2
G K
6 56148 IGE *

30. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
.3 . . .
GPUs
Tesla K80Tesla K80
- 0
Prometheus
https://prometheus.io/
GPU monitoring
script
3 B
: 3 / B
Grafana
(https://grafana.com/, UI)
: 3 0:
Managements
: . : B
: :

31. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
- . . 3

32. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
- . . 3
GPU Clocks GPU Temperature

33. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
- . . 3
GPU Clocks GPU Temperature

34. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• : : : :
• . : .
• . : :3
• A 4 A ?4 4 ? ? - (
: :
Storage
? 3 ) ?
4 4
Storage
? 3 ) ?
Storage
? 3 ) ?
4 4 4
AI Framework Stacks
Chainer MN Chainer Other AI Framework e.g. TensorFlow)
4 D
- ? C (
4 4 A 43

35. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• 3 . 3
• 3 3 3 3 5 3 35 5
3 3 3
(( )
(( ( (
( (

36. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• . 2 223
. 1 F D ( ) .3- F
1 D 1 D A /A 1D 1 D D D 6 D
11) .3-
11) 1(1
)) 1(1
2 - 3 3 . 2
11) .3-
11) 1(1
)) 1(1
H
2 6 D ( ) 6 . 1 D D
1 D 6 1 D 1
1 D 3

37. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
3 . .
3 - / -C7 (7 7 3 3 - / 3
-C7 )- ) 3 / 3 7:3 D 3 -C7
:: - . 3 D 3 / 7 7:3 / 3
-C7 : 3
7 7 C7
Swift FS (File-System-in-User-Space, FUSE)
3 / C7 :/ 3

38. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
. 3
) 1 A 1 ) 1 : 31
OS
Swift FS (File-System-in-User-Space, FUSE)
AI Framework Stacks
Chainer MN Chainer
) 1 : 31 ( A
1 18 ( : ) ( (
Swift Client SDKs

39. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• + : . :
• 3 3
• 9 33 3
• 3
:
• 3
• 9 . 3 3 9
33 . 3 3
: . . -

40. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
•
• 4 / 0 / /
• 4 4 / /
• 0 4
•
• 4 / 4 /
• 4 / 4 0 /
,3 .- .-

41. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
•
• .4 8 4 : ? F A F 4 8 64?8 F ?8 F8
A? A: C8A F46 I F
• A 8D4F A I F 64?8 F 8 D84 A FD F8
46 A8 ?84DA A:
• 64A ?84DA 4F FFC : !:? D
• : : ,
: 3 :
• /88 F ?84DA D8!!!
• 8 4A ? F A 4 8 4? 4668 8FI88A
?8 F8 4A 86F 1-
, 3 - , 3.
1?84 8 4D8 6 F 8 : ? F A

42. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
• 4--, : / - - -
. - . - - -4: 4 -
• /- - 4 .. :/- /- . -:-
-4 4 -4: - . 4
/ , - :- -:
• 4 .4 2 4
• 2 4 4 .4 4
• 4 -
- 44 : - : - 4,-
. -
•
- 4 - 4-,

43. Copyright©2018 NTT Corp. All Rights Reserved.
NTT Confidential
> < @ > > <
> C > C > . >>
4 < > 4 < . >>
3 > > 3 . >>