The document discusses using Docker on the K computer supercomputer. Docker could help with compiling and running software by providing consistent software environments. At compile time, Docker containers could be used on login nodes to install cross-compilers and compile software in a consistent environment. At runtime, Docker could potentially be installed on K computer compute nodes, but the operating system kernel would need to be updated from 2.6.25 to 3.8 to support Docker, requiring patching 84 files. Issues include compiler licenses and administrative policies. A Docker-based IaaS approach is proposed to allow compiling software in containers on a server to overcome dependency and installation challenges.

1. Docker and
the K computer
Bryzgalov Peter @ HPC Usability Research Team
RIKEN AICS
petr.bryzgalov@riken.jp
2015.08.19 Docker Meetup Tokyo #5

2. K computer

3. K computer
Nodes > 80 000
Cores > 700 000
Linpack Performance 10,510 TFlop/s

4. Docker merits for HPC
• Runtime — customise software stack, easy to
test new software, more efficient resource
management, administrators do not have to
worry about security patches for user-space
software, etc.
• Compile time — eliminate dependencies on
compilers and libraries versions, etc.
4

5. Docker merits for HPC
• Runtime — customise software stack, easy to
test new software, more efficient resource
management, administrators do not have to
worry about security patches for user-space
software, etc.
• Compile time — eliminate dependencies on
compilers and libraries versions, etc.
5

6. Docker at Compile time
• Install cross-compiler in container
6

7. Docker at Compile time
• Install cross-compiler in container
Why cross-compiler?
7

8. K architecture
• Login nodes and Compute nodes
• Compilation on Login nodes
• Execution on Compute nodes
8

9. K architecture
• Login nodes and Compute nodes
• Compilation on Login nodes
• Execution on Compute nodes
• Login nodes — x86_64
• Compute nodes — sparc64fx
9

10. Docker at Compile time
Done :
• Installed cross-compilers in container.
• Developed system for using it online.
10

11. Docker at Compile time
Workflow :
• User runs a script instead of “make”
command on his computer.
• Source code folder automatically mounted
into container with compilers.
• “make” command automatically executed in
container.
11

12. Docker at Compile time
1.Container with cross-compilers runs on server.
12

13. Docker at Compile time
2.Source code folder automatically mounted
into container.
13

14. Docker at Compile time
3.“make” command automatically executed in
container.
14

15. Docker at Compile time
Result :
• Binaries are built and saved on user
computer.
15

16. Docker at Compile time
Issues :
• Compiler license.
• Customise software stack (instal third-party
software in container).
16

17. Docker merits for HPC
• Runtime — customise software stack, easy to
test new software, more efficient resource
management, administrators do not have to
worry about security patches for user-space
software, etc.
• Compile time — eliminate dependencies on
compilers and libraries versions, etc.
17

18. Docker at Runtime
• Install Docker on Compute nodes
18

19. Docker at Runtime
• Install Docker on Compute nodes
• K compute nodes:
19

20. Docker at Runtime
• Install Docker on Compute nodes
• K compute nodes:
20

21. Docker at Runtime
• Install Docker on Compute nodes
• K compute nodes:
20
CPU sparc64fx
OS kernel 2.6.25

22. Docker at Runtime
• Install Docker on Compute nodes
• K compute nodes:
• Docker requires 3.8 (3.10 for latest version)
21
CPU sparc64fx
OS kernel 2.6.25

23. Update OS kernel
on Compute Nodes
• 2.6.25 -> 3.8
22

24. Update OS kernel
on Compute Nodes
• 2.6.25 -> 3.8
• OS kernel on K CN:
vanilla 2.6.25.8 + Fujitsu patch
• Apply Fujitsu patch to vanilla 3.8
23

25. Patch vanilla kernel
• Take a new vanilla kernel source code and
apply Fujitsu patch
24

26. Patch vanilla kernel
• Take a new vanilla kernel source code and
apply Fujitsu patch
• Patch fails starting from 2.6.26
25

27. Patch vanilla kernel
• How many files need to be fixed?
26

28. Patch vanilla kernel
27

29. Patch vanilla kernel
28
84f i l e s

30. Patch vanilla kernel
29
We need
help!

31. Maeda Toshiyuki Team leader
Bryzgalov Peter Researcher
Thank you!
HPC Usability Research Team
www.hpcu.aics.riken.jp
comp.photo777.org
petr.bryzgalov@riken.jp
This presentation has more slides but I don’t have time to show them.

32. Docker merits for HPC
• Runtime — customise software stack, easy to
test new software, more efficient resource
management, administrators do not have to
worry about security patches for user-space
software, etc.
• Compile time — eliminate dependencies on
compilers and libraries versions.
31

33. Docker at Compile time
K workflow
• Edit Source code
• Cross-compile on Login Nodes
• Execute on Compute Nodes
32

34. Docker at Compile time
K workflow
• Edit Source code
• Cross-compile on Login Nodes
• Execute on Compute Nodes
Dependencies
• Compilers, libraries, … — one environment for all, it changes
over time, application that compiles now may not compile in
the future (like ABYSS).
33

35. Docker at Compile time
K workflow
• Edit Source code
• Cross-compile on Login Nodes
• Execute on Compute Nodes
Dependencies
• Compilers, libraries, … — one environment for all, it changes
over time, application that compiles now may not compile in
the future (like ABYSS).
33
Cross-compile in Docker container

36. Compile in container
• Run container on User Computer
• Run container on Login Node
• Run container on Server
34

37. Compile in container
• Run container on User Computer
• Run container on Login Node
• Run container on Server
34
Compiler licence limitations

38. Compile in container
• Run container on User Computer
• Run container on Login Node
• Run container on Server
34
Compiler licence limitations
“K” administration policies

39. Compile in container
• Run container on User Computer
• Run container on Login Node
• Run container on Server
35

40. Run compiler on server
in Docker container
Requirements
• Multiple environments (containers)
• Multiple users
• One machine
• Limited resources — stop containers when they are not used
Tasks
• Copy source code into containers
• Copy binaries from containers
36

41. Docker-based IaaS
Overview
• One host multi tenant IaaS
• One user — one container
• Users login with SSH
• Mount local folders into containers with SSHFS
37
github.com/pyotr777/dockerIaaSTools

42. Docker-based IaaS
Overview
38
github.com/pyotr777/dockerIaaSTools

43. Docker-based IaaS
Application
Challenge
• K-scope — tool for Fortran program static analysis.
• Depends on Omni XMP compiler front-end, which produces
AST (XML files).
• Omni XMP depends on many packages and is hard to install.
39
omni-compiler.org/xcalablemp.html
www.aics.riken.jp/ungi/soft/kscope
github.com/pyotr777/dockerIaaSTools

44. Docker-based IaaS
Application
Our solution
• Script that mounts local source code folder in container and
executes “make” command inside the container.
• K-scope executes the script instead of make command.
40
github.com/pyotr777/dockerIaaSTools

45. Docker-based IaaS
Application
Our solution
41

46. Docker-based IaaS
Application
Our solution
• Script that mounts local source code folder in container and
executes “make” command inside the container.
• K-scope executes the script instead of make command.
• After script finishes Intermediate code files are in local source
folder — same effect as calling “make” command locally.
42
github.com/pyotr777/dockerIaaSTools