1. Image: NASA
CLAW Code Manipulation for Performance Portability
PASC’16 - MS03 Code Generation Techniques for HPC Earth Science Applications
8th June 2016
Valentin Clement
valentin.clement@env.ethz.ch

2. Valentin Clement 8th of June 2016 - PASC’16 2
Summary
• Performance portability problem
• CLAW 1st approach: Low level transformation
• Column model abstraction
• CLAW 2nd approach: High level abstraction
• CLAW Compiler
• CLAW current status and next steps

3. Valentin Clement 8th of June 2016 - PASC’16 3
Hybrid supercomputer (CPU/GPU)

4. Valentin Clement 8th of June 2016 - PASC’16 4
COSMO to the Hybrid supercomputers
Initialization
Cleanup
Boundary conditions
Physics
Dynamics
Data assimilation
Halo-update
Diagnostics
Input / Output
Δt
à OpenACC port
à OpenACC port
à C++ / DSL rewrite
à Mixed OpenACC / CPU
à Communication library (GCL)
à OpenACC port
à Mixed OpenACC / CPU
Interface
Interface
Copy to accelerator

5. Valentin Clement 8th of June 2016 - PASC’16 5
COSMO radiation example
!$acc parallel loop
DO j=1,nproma
!$acc loop
DO k=1,nz
CALL fct()
! 1st loop body
! 2nd loop body
! 3rd loop body
END DO
END DO
!$acc end parallel
DO k=1,nz
CALL fct()
DO j=1,nproma
! 1st loop body
END DO
DO j=1,nproma
! 2nd loop body
END DO
DO j=1,nproma
! 3rd loop body
END DO
END DO
CPU optimal GPU optimal
Iteration over 2 dimensions (block, column level)

6. Valentin Clement 8th of June 2016 - PASC’16 6
Performance portability: COSMO Radiation
Runtime[s]
0
0.3
0.6
0.9
1.2
Code restructured for CPU Code restructured for GPU
CPU GPU
-30%
-35%

7. Valentin Clement 8th of June 2016 - PASC’16 7
Performance portability: past situation
#ifndef _OPENACC
DO k=1,nz
CALL fct()
DO j=1,nproma
! 1st loop body
END DO
DO j=1,nproma
! 2nd loop body
END DO
DO j=1,nproma
! 3rd loop body
END DO
END DO
#else
!$acc parallel loop
DO j=1,nproma
!$acc loop
DO k=1,nz
CALL fct()
! 1st loop body
! 2nd loop body
! 3rd loop body
END DO
END DO
!$acc end parallel
#endif
• Multiple code paths
• Hard maintenance
• Error prone
• Domain scientist have to
know well each target
architectures
CPU
GPU

8. Valentin Clement 8th of June 2016 - PASC’16 8
Performance portability: optimizations
• Loop fusion
• Loop reordering
• Loop extraction
• Loop hoisting
• Caching
• On the fly computation
• Array notation to do statement
First step: COSMO radiation/HAMMOZ
CPU optimal —> GPU optimal

9. Valentin Clement 8th of June 2016 - PASC’16 9
CLAW: first steps —> Low level transformations
!$acc parallel loop
!$claw loop-interchange
DO k=1,nz
!$claw loop-extract fusion
CALL fct()
!$claw loop-fusion
!$acc loop
DO j=1,nproma
! 1st loop body
END DO
!$claw loop-fusion
!$acc loop
DO j=1,nproma
! 2nd loop body
END DO
!$claw loop-fusion
!$acc loop
DO j=1,nproma
! 3rd loop body
END DO
END DO
!$acc end parallel
!$acc parallel loop
DO j=1,nproma
!$acc loop
DO k=1,nz
CALL fct()
! 1st loop body
! 2nd loop body
! 3rd loop body
END DO
END DO
!$acc end parallel
CLAW compiler
• CLAW v0.2
• 8 low level transformations
• CPU optimal —> GPU optimal

10. Valentin Clement 8th of June 2016 - PASC’16 10
Hard to use - Too many directives…

11. Valentin Clement 8th of June 2016 - PASC’16 11
CLAW Higher level abstraction for domain scientist
Separation of concerns
• Domain scientists focus
on their problem (1
column, 1 box)
• Dependency only in k
levels
• CLAW compiler produce
code for each target and
directive languages

12. Valentin Clement 8th of June 2016 - PASC’16 12
CLAW abstraction column/box model
SUBROUTINE compute_col(q,t,nz)
INTEGER, INTENT(IN) :: nz
REAL, INTENT(INOUT) :: q(:),t(:)
INTEGER :: k
REAL :: c
!$claw define dimension j(1:nproma) &
!$claw parallelize
c = 5.345
DO k = 1, nz
t(k) = c * k
q(k) = t(k) * c
END DO
END SUBROUTINE column_model
clawfc -d=openacc -t=gpu -o mo_col_gpu.f90 mo_col.f90
clawfc -d=openmp -t=cpu -o mo_col_cpu.f90 mo_col.f90
original
code
GPU
optimized
CPU
optimized
• Column model code
encapsulated in a
subroutine
• Agnostic of any
architecture
• Let CLAW manage the
parallelization over the
domain
clawfc -d=openmp -t=mic -o mo_col_mic.f90 mo_col.f90
MIC
optimized

13. Valentin Clement 8th of June 2016 - PASC’16 13
CLAW compiler
• Based on the OMNI Compiler
• Source-to-source translator
• Open source under the BSD license

14. Valentin Clement 8th of June 2016 - PASC’16 14
CLAW compiler
Sets of programs/libraries to build source-to-source compilers for C
and Fortran via an XcodeML intermediate representation
-Used to implement XcalableMP (abstract inter-node communication), XcalableACC (XMP +
OpenACC), OpenMP (implementation for C and Fortran), OpenACC (C implementation only)
Development team
– Programming Environments Research Team from the RIKEN Advanced
Institute for Computational Sciences, Kobe, Japan
– High Performance Computing System Lab, University of Tsukuba, Tsukuba
– http://www.omni-compiler.org
– https://github.com/omni-compiler
OMNI Compiler

15. Valentin Clement 8th of June 2016 - PASC’16 15
CLAW Current status and next steps
• Low-level transformations implemented
• COSMO Radiation using them
• Heading towards high-level abstractions and
automatic transformations
• Working with examples from:
• COSMO
• ICON RRTMGP
• ECHAM-HAMMOZ

16. Valentin Clement 8th of June 2016 - PASC’16 16
Resources
https://github.com/C2SM-RCM

17. Valentin Clement 8th of June 2016 - PASC’16 17
valentin.clement@env.ethz.ch
Involved persons: Jon Rood (C2SM), Sylvaine Ferrachat (ETH Zurich), Will Sawyer
(CSCS), Oliver Fuhrer (MeteoSwiss), Xavier Lapillonne (MeteoSwiss)