Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

An Overview of the IHK/McKernel Multi-kernel Operating System


Published on

By Balazs Gerofi, RIKEN Advanced Institute For Computational Science

RIKEN Advanced Institute for Computation Science is in charge of leading the development of Japan's next generation flagship supercomputer, the successor of the K. Part of this effort is to design and develop a system software stack that suits the needs of future extreme scale computing. In this talk, we focus on operating system (OS) requirements for HPC and discuss IHK/McKernel, a multi-kernel based operating system framework. IHK/McKernel runs Linux with a light-weight kernel (LWK) side-by-side on compute nodes with the primary motivation of providing scalable, consistent performance for large scale HPC simulations, but at the same time to retain a fully Linux compatible execution environment. We provide an overview of the project and discuss the status of its support for ARM architecture.

Balazs Gerofi Bio
Research Scientist at RIKEN Advanced Institute For Computational Science.


For more info on The Linaro High Performance Computing (HPC) visit

Published in: Technology
  • Be the first to comment

  • Be the first to like this

An Overview of the IHK/McKernel Multi-kernel Operating System

  1. 1. An Evaluation of EasyBuild for Open Source Software Deployment Arm Environment 14:00 - 14:25 12th December 2017 - Arm HPC Workshop, Akihabara System Software Development Team RIKEN Advanced Institute for Computational Science Takahiro OGURA 1
  2. 2. Agenda 2 1. Problems you might encountered when building/executing OSS 2. How EasyBuild helps solving these problems 3. Structure of EasyBuild 4. Testing EasyBuild on Arm architecture 5. Creating recipes for EasyBuild 6. Conclusions
  3. 3. Problems you might encounter when building/executing OSS 3  If the OSS (= HPC software stack not in Linux distro., such as PETSc) you want use is not installed on the system you have to build it by your own. Problems you might encountered:  It is time-consuming to obtain all the OSSs needed to build the target OSS (ex. PETSc requires 40 OSSs to build)  It is confusing and time-consuming to set/switch the environmental variables  Environment dependent build settings  Lack of architecture support other than x86 (e.g., SPARC and Arm) often requires manual patching  Similarly for vendor-specific compilers and/or cross compilers
  4. 4. How to solve OSS building/executing problem 4  Similar problems can also occur on post K computer  Linaro’s effort solves many of them, but won’t cover all  Operators should install OSSs instead of individual users  Operator should share know-hows  CEA in France is using EasyBuild to manage OSSs on x86 Xeon clusters to ease the OSS building/execution  EasyBuild: Framework for building and managing OSS commonly used in HPC  Step 1. Understand EasyBuild’s framework and how it works  Step 2. Test EasyBuild on Arm architecture machine  Step 3. Test EasyBuild on cross compiling environment with vender-provided compiler  Currently at Step 2.
  5. 5. How EasyBuild works with PETSc 5  40 OSSs needed to build PETSc 3.7.3  Some of these provided by RPMs  Some of these provided by source (tar.gz) from the web sites  Some of these provided by github repository Autoconf-2.69 Automake-1.15 Autotools-20150215 Autotools-20150215-foss-2016a binutils-2.25 Bison-3.0.4 Boost-1.61.0 bzip2-1.0.6 CMake-3.4.3 CMake-3.5.2 FFTW-3.3.4 FIAT-1.6.0 flex-2.5.39 GCC-4.9.3-2.25 GCCcore-4.9.3 GMP-6.1.0 gompi hwloc-1.11.2-GCC-4.9.3-2.25 Hypre-2.11.1 libreadline-6.3 libtool-2.4.6 M4-1.4.17 METIS-5.1.0 mpmath-0.19 ncurses-6.0 numactl-2.0.11 numpy-1.8.2 OpenBLAS-0.2.15 OpenMPI-1.10.2 ParMETIS-4.0.3 Python-2.7.11 ScaLAPACK-2.0.2 ScientificPython-2.9.4 SCOTCH-6.0.4 SQLite-3.9.2 SuiteSparse-4.5.3 sympy-1.0 Tcl-8.6.4 Tk-8.6.4 zlib-1.2.8
  6. 6. How EasyBuild collects dependences 6  Execute the following command: $ eb PETSc-3.7.3-foss-2016a-Python-2.7.11.eb  Then download, git clone the source and build by following EasyBuild recipes  However, EasyBuild does not support RPMs...  You have to build and install OSSs even RPMs are provided officially
  7. 7. Setting environmental variables 7  You need to set the environmental variable such as INCLUDE_PATH, LD_LIBRARY_PATH, PATH and etc. to build the OSS  You need to set the environmental variable such as PATH to execute the OSS  EasyBuild creates environment modules automatically  Environment modules allows users set and unset the environmental variables by a single command: $ module load xxxx
  8. 8. How EasyBuild works 8 1. execute eb command. Specify Easyconfig file. $ eb PETSc-3.7.3-foss-2016a-Python-2.7.11.eb PETSc-3.7.3-foss-2016a-Python-2.7.11.eb Easyconfig (text file) 2. fetch sources 3. unpack sources 4. apply patches 5. configure 8. sanity check 7. install 6. build/make Easyblock (Python2 Script) Internet Download git clone Environment Modulefiles Include headers Executable Binary files Executable Binary files Executable Binary files Library Binary files Include headers Functions from EasyBuild Framework
  9. 9. 9  EasyBuild framework  Python2 API set class library to use EasyBuild  Python3 in not supported yet  Easyblock  Script written in Python2 using EasyBuild framework  Need Easyblock for each OSS  Easyconfig  Text setting file which Easyblock reads  Modify when changing building settings such as configure options  Written in Key = Value format Structure of EasyBuild For more info:
  10. 10. 10  A part of classes of EasyBuild framework  easybuild.framework.easyblock.EasyBlock  Basic class for Easyblock building and install software load Enviroment Module  easybuild.framework.easyconfig.easyconfig.EasyConfig  Class to read Easyconfig  easybuild.framework.easyconfig.easyconfig.copy_easyconfigs  Class to copy Easyconfig file to designated path  easybuild.framework.easyconfig.easyconfig.copy_patch_files  Class to copy patch file to designated path  easybuild.framework.easyconfig.default.get_easyconfig_parameter _default  Class to get Easyconfig values. EasyBuild framework For more info:
  11. 11. 11 Major settings in Easyconfig Setting Item Name(Key) Examples of value Description Easyblock EB_PIP Name of Easyblock file. name PIP Name of software. version 0.1 Version of OSS. homepage Web site URL of OSS (Top page.) description PiP is a library to … Description of OSS. toolchain {'name': 'GCC', 'version': '4.9.3-2.25'} Dependence OSS name. builddependencies ('PIP-glibc', '2.17') Dependence OSS name when building target OSS. osdependencies ('nss-devel', '') Name of package which must installed in OS when building target OSS. sources ['%(name)s.tar.gz'] Name of source file. source_urls [' wer)s/%(version)s/'] URL of source code tar ball. patches ['PETSc_ranlib-fix.patch'] Name of patch file if needed. configopts/buildopts/inst allopts 'CC=gcc CXX=g++ --enable-add-ons' options for configure/make/make install runtest 'check ' setting for test after make. sanity_check_paths {'files': "bin/hello" } Check file/directory presence. For more info:
  12. 12. 12 Step2. Test EasyBuild on Arm architecture machine Testing environment Hardware • SoftIron OverDrive 3000 • CPU: AMD Opteron A1170 (2GHz 8 cores) • memory: 16GB • Network:10Gbase-T * 2 • Disk: 1TB * 1 Software • OS: CentOS 7.3.1611 • Kernel: 4.5.0-29.el7.aarch64 • Python: 2.7.5 • Lua: 5.1.4 • Lmod: 7.6 • GCC: 4.8.5 • EasyBuild: 3.4.0, 3.4.1 Requirement for EasyBuild • Linux OS • Python 2.6 or higher • Lmod or Tcl ※ Lmod is suggested • Lua • lua-filesystem • lua-posix • lua-devel • libreadline • ncurses • C/C++ Compiler (GNU gcc etc.) For more info:
  13. 13. 13  Installing Lmod  No RPM for Lmod is available for Aarch64 So, we built it from the source code Installing EasyBuild(1/2) For more info: $ curl -LO $ tar xfvj Lmod-7.6.tar.bz2 && cd Lmod-7.6 $ ./configure --prefix="<install path>" && make install • Add PATH settings to .bashrc • Test Lmod $ lmod --version Modules based on Lua: Version 7.6 (7.6-3-gbbb0e9b7) 2017-07-24 14:07 -05:00 by Robert McLay export PATH=$HOME/lmod/7.6/libexec:$PATH
  14. 14. 14  Installing EasyBuild  Set environmental variables  Download bootstrap script  execute install script  Set Environment Modules PATH and test eb command Installing EasyBuild(2/2) For more info: For more info : $ export EASYBUILD_PREFIX=$HOME/.local/easybuild $ curl -O d/scripts/ $ python $EASYBUILD_PREFIX $ module use $/EASYBUILD_PREFIX/modules/all $ module load EasyBuild $ eb --version This is EasyBuild 3.4.1 (framework: 3.4.1, easyblocks: 3.4.1) on host ***
  15. 15. 15  PIP is a software being developed by our team  PIP requires glibc to build and execute  Make 2 sets (for glibc and PIP) of Easyblock and Easyconfig Building PIP on Arm/Creating recipes for PIP Easyconfig PIP-0.1.eb Easyblock Easyblock Easyconfig PIP-glibc-2. 17.eb
  16. 16. 16  = 100 lines, = 10 lines  Rules for Easyblock classes name to add  Name of a new Python class for each EasyBlock must start from “EB_” and the target software name.  Simply execute encode_class_name() to name new Python class  Rules for Easyblock file name.  To name a new Easyblock file simply execute get_module_path(). Making Easyblock for PIP For more info: >>> from import encode_class_name >>> encode_class_name('PIP-glibc') 'EB_PIP_minus_glibc’ ←New class name >>> from from easybuild.framework.easyconfig.easyconfig import get_module_path >>> get_module_path('PIP-glibc') 'easybuild.easyblocks.pip_glibc’ ←New Easyblock file name
  17. 17. 17 (Easyblock for PIP) class EB_PIP(ConfigureMake): # Class definition (Inherit ConfigureMake class) def configure_step(self): # Function definition # statically link to PIP-glib if it is a (build) dependency pipglibc = get_software_root('PIP-glibc') # Get PIP-glibc path if pipglibc: glibc_libs = os.path.join(pipglibc, "lib") # Get path of lib directory for PIP-glibc # Set configure options cmd = './configure --prefix="%s" --with-glibc-libdir="%s"' % (self.installdir, glibc_libs) # run configure run_cmd(cmd, log_all=True, simple=True) # Execute configure and make install
  18. 18. 18 (Easyblock for PIP-glibc)(1/2) <snip> import as env from easybuild.easyblocks.generic.configuremake import ConfigureMake # import configuremake module from import EasyBuildError # import build_log module from import run_cmd # import run module class EB_PIP_minus_glibc(ConfigureMake): # Inherit ConfugureMake and define EB_PIP_minus_gbibc def create_dir(self, dirname): # function to create new dir """ Create a dir to build in. """ dirpath = os.path.join(self.cfg['start_dir'], dirname) try: os.mkdir(dirpath) os.chdir(dirpath) self.log.debug("Created dir at %s" % dirpath) return dirpath except OSError, err: raise EasyBuildError("Can't use dir %s to build in: %s", dirpath, err) (go to next page)
  19. 19. 19 (continue from previous page) def run_configure_cmd(self, cmd): # Get a value “preconfigopts” from Easyconfig (out, ec) = run_cmd("%s %s" % (self.cfg['preconfigopts'], cmd), log_all=True, simple=False) if ec != 0: raise EasyBuildError("Command '%s' exited with exit code != 0 (%s)", cmd, ec) unknown_re = re.compile("WARNING: unrecognized options") unknown_options = unknown_re.findall(out) if unknown_options: raise EasyBuildError("Unrecognized options found during configure: %s", unknown_options) def configure_step(self): # overwrite configure_step function configopts = '' # configopts is only used in first configure self.configopts = self.cfg['configopts'] # Get a value of “configopts” from Easyconfig self.create_dir(self, "glibcobj") # actual configure, but not on default path # Use value from Easyconfig as a configure option cmd = '../configure --prefix="%s" %s %s' % (self.installdir, self.configopts, configopts) self.run_configure_cmd(cmd) <snip> for PIP-glibc)(2/2)
  20. 20. 20 Easyconfig for PIP/PIP-0.1.eb(1/2) For more info: KEY=VALUE Description Easyblock = 'EB_PIP' Use EB_PIP as an Easyblock name = 'PIP' Environment Module name version = '0.1' Environment Module version homepage = '' Web site of OSS description = "PiP is a library to create processes in process virtual address space implemented at user-level." Description for the software Used as Environment Module description toolchain = {'name': 'dummy', 'version': ‘’} Tool-chain to use builddependencies = [ ('PIP-glibc', '2.17'), ] Define dependency to build PIP sources = ['%(name)s.tar.gz’] Path to the source code To Download the source code define “source_urls” (Go to next page)
  21. 21. 21 Easyconfig for PIP/PIP-0.1.eb(2/2) KEY = VALUE Description sanity_check_paths = { 'files': [ "bin/hello", <snip> ], 'dirs': [ "bin/", <snip> ], } Check presence of directories and files runtest = 'check ' Before install run make check sanity_check_commands = [('%(installdir)s/bin/hello', ‘’), ] How to check the presence of directories and files postinstallcmds = [('%(installdir)s/bin/piplnlibs’), ] Specify command after install_step(make install) moduleclass = 'lib' Define software type Can check by “$ eb –moduleclasses” command For more info:
  22. 22. 22 Run PIP-0.1_Aarch64.eb $ eb PIP-0.1_Aarch64.eb == temporary log file in case of crash /tmp/eb-AviEmJ/easybuild-_KRKyK.log == processing EasyBuild Easyconfig /home/user/PIP-0.1_Aarch64.eb == building and installing PIP/0.1... == fetching files... == creating build dir, resetting environment... == unpacking... == patching... == preparing... == configuring... == building... == testing... == installing... == taking care of extensions... == postprocessing... == sanity checking... == cleaning up... == creating module... == permissions... == packaging... == COMPLETED: Installation ended successfully == Results of the build can be found in the log file(s) /home/user/easybuild/software/PIP/0.1/easybuild/easybuild-PIP-0.1-20171030.16383 9.log == Build succeeded for 1 out of 1 == Temporary log file(s) /tmp/eb-AviEmJ/easybuild-_KRKyK.log* have been removed. == Temporary directory /tmp/eb-AviEmJ has been removed. $ module load PIP/0.1 $ hello Hello (PID=3038)!!
  23. 23. Conclusions 23  EasyBuild has potential for solving typical OSS building/executing problems  However, to learn/understand EasyBuild framework it takes almost a month (for Python beginners) and the EasyBuild framework needs more functions such as support for RPM, cross-compiling features, etc.  Once we understood the EasyBuild framework, it took only a week to create Easyblock and Easyconfig for our software  We will further evaluate whether EasyBuild can adopt cross- compiling environments with vendor-provided compilers
  24. 24. 24 Thank you for your attention! Any questions or comments?