Continuous Integration and Delivery of Research Applications

Continuous Integration and
Delivery of
Research Applications
Bruce Becker
bbecker@csir.co.za
Coordinator, AAROC
Meraka Institute, CSIR
Fanie Riekert
RiekertSJPK@ufs.ac.za
Senior Systems Engineer
HPC Unit, UFS
Steps towards an Open e-Science Commons

Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za
Outline
●
Compuational Infrastructure: It's a jungle out
there
●
How does research actually get done ? A
reminder of the scientific method in the 3rd and
4th paradigm
●
How do applications get onto an infrastructure ?
●
What are the issues ?
●
What are we doing about it ?
●
What can you do about it ?

Bruce Becker: Coordinator, SAGrid | bbecker@csir.co.za | http://www.sagrid.ac.za3
Functional
Sites
Core Services
6 sites
~2300 CPU cores
~100 TB perm. Storage
~17 permanent staff
Proposed extensions
So, we built this...

And others are too

Universite'
Gaston
Berger
(Saint-Louis)
Kwame Nkrumah
University of
Science
And Technology
(Kumasi, Ghana)
Nsukka
University
(Nsukka)
http://cds.cern.ch/record/1476
353
220 servers, 30
routers
And others are too

And it is part of this
blue: EGI council members and EGI-InSPIRE
partners
green: integrated resource infrastructure providers
purple: peer resource infrastructure providershttp://www.egi.eu/infrastructure/

Great, right ?!

It's (apparently) a jungle out there.
http://en.wikipedia.org/wiki/Jungle_computing
Seinstra, F. J., Maassen, J., van Nieuwpoort, R.~V., et al. 2011,
Grids, Clouds and Virtualization, 167
Jungle Computing: Distributed Supercomputing beyond Clusters, Grids, and Clouds

Where does your code run ?
●
On your laptop ?
●
Congrats ! Uh oh, you just rand out of X
●
You want to do what !?
●
Local cluster ?
●
hm. different setup... <hack hack>
hey it's working ! … kinda.
●
Permissions, environment,
data staging, etc.
●
CHPC ?
●
There's more than one
●
ALL the Optimisations !!

Are you even talking about the
same application ?
●
Apart from performance, are you sure that you
are getting the same results on different
architectures and platforms ?
●
Is it even worth the effort ?

The problem

What users want
Amazing infrastructure
Some users want highly
varied, modular
application selection
Vertically integrated
Highly specialised
applications
Highly trained supportHighly trained support

What users get sometimes

Current Methods
Are you a legit
scientist ?
Ok, sounds good
what applications
are you running ?
We can run
almost anything !
Hi, I heard about
AfricaGrid – sounds
grrrrreat ! Can I
Use it ?
I've got beakers
…
And glasses !
Awesome ! Well, first we do some
preprocessing with RaNDo, then we
run a massively parallel pipeline with
XBOT, then blah and blah then blah
blah blah and blah, we also need blah
blah and sometimes our colleagues
from FarAwayU need to access the
data with StrangeProtocolx1, and of
course blah blah, not to mention blah
…..
We'll see
what we
can do

Actual workflow
●
Find the source code
●
Unpack, ./configure && make … Error 1 missing x.h
●
… hack hack hack … hm. Missing dependencies
●
Recursively* determine the dependencies**
●
*Yes, recursively means curse, curse then curse again.
●
**The actual dependencies, not the ones that were written on the
web page from 5 years ago.
●
… mutter mutter …
●
???? Why are you compiling now !?

Actual communication
Hey, how's it going ?
you done with my porting yet ?
You done yet ?
You done ?
…. bloody IT people !
Have you actually seen this code ?
I just...
It looks like Cthulhu's beard....
Ok, I got it to compile,
but only with if you
type with your left toes.
AAArrrgh, I'm a scientist,
not an IT people !

How many applications are there ?
Natural Sciences
Other
Medical and Health Sciences
Support Activities
Engineering and Technology
Agricultural Sciences
Humanities
Social Sciences
http://appdb.egi.eu

Way too many...
(and these are just the ones we
know about)
Biological sciences
Bioinformatics
Biochemistry and molecular biology
Structural biology
Biophysics
Genetics and heredity
Cell biology
Evolutionary biology
Physical sciences
Astronomy
Astroparticle physics
Astrophysics
High energy physics
Fusion
Plasma physics
Fluid Mechanics
Condensed matter physics
Mathematical physics
Medical physics
Optics
Physics
Acoustics

There's gotta be a better way
Let's give researchers a
”better deal”

• Hypothesis 1:
– It always comes down to an application
• At the end of all first-contact you get a list of applications
that users want
– Let's focus on that, shall we ?
• Hypothesis 2:
– No software is an island
• Everything has dependencies, even if it's just glibc or
libgfortran
– Let's make those explicit, shall we ?
Compuatational Infrastructure
What is it good for ?

• Hypothesis 3:
– Applications need an environment
• Compiler ?
• Architecture ?
• Dependencies ?
• Middleware ?
• Etc....
– Let's make those explicit shall we ?

• Hypothesis 4:
– there is more than one environment.
• Nothing is going to ever be really useful if it can only run in
one place.
• Nobody wants to be forced into a single environment
(grid/HPC/cloud, etc)
– Let's try to simulate as many target environments (sites)
as we can, shall we?

●
Hypothesis 5:
●
Solutions decay
– The solution – as well as the dead ends – are part of the
process.
– If it's not in a version-controlled repository, it's no good.
– If it's not executable (in a wiki/document, etc), then you
have to convert it back into something a computer
understands – why not just leave it in executable format ?
●
Let's recognise that solutions decay and code for the
future, shall we ?

Humans need not apply
(for certain tasks)
●
Hypothesis 6:
●
Most of the work of porting and supporting applications
can be automated
– Reduce the load on the user and the operator, by automating
common tasks
●
Let's:
– Integrate the various software, middleware, service and
infrastructure components as far as possible,
– Keep it Open
●
… shall we ?

●
Hypothesis 6 (Corollary) : You're in the way

●
Hypothesis 7:
●
This stuff's not hard
– Continuous Integration has been around for a long time
– If it can be done by hand, it can be scripted; if it can be
scripted, it can be automated properly.
●
Let's
– adopt widely-used tools and methodologies, and
– hey, how about we use the web to communciate
(you know, like we do with EVERYTHING else ?)

Let's work on a better deal...
●
How do we lower the barrier to entry to the grid or
cloud infrastructure ?
●
How can the application expert prove to the resource
provider that the application will actually run on the
execution environment of the site ?
●
How can we manage the lifecycle of applications across
multiple versions, architectures, configurations ?
●
How can we ensure that once applications are ”certified”,
they are actually available on as many sites as possible ?

Go to Bootcamp
http://software-carpentry.org/bootcamps

What are we actually doing ?
●
Put the ball in the developer/community/user's court
●
You tell us how to build your application
●
Provide a simple interface to porting pipeline
●
Provide focussed, goal-oriented support when needed.
●
Communication has a context and is directly linked to the issues at
hand
●
The entry point is open
●
Extensible and reproducible
●
Publish and preserve the entire toolchain
●
It's a real collaboration now – with code !
●
Can we say the big S word ? (not the 4-letter word - SUSTAINABLE)

Tools of the Trade (1)
Waffle.io
Project
planning
Slack.com
Team collab

What's the plan, Stan ?
https://waffle.io/aaroc/devops

Typical workflow and roles
Testingmatrix
Defines relevant
tests in Jenkins
Writes code to
pass required tests
Dev/Stage env.Application
developer
Infrastructure
expert
Reads description
of execution environment tests
Promote a build
to CVMFS

My Github brings all the hackers to
the backyard

Social coding
Identify yourself, create a branch
Jenkins will help you test your code
at every commit
Contribute back, open a pull request
Discuss contrbutions
https://guides.github.com/introduction/flow/index.html

Get credit for your work
Bruce Becker, Marco Fargetta, pkoro, Pavlos Daoglou, Chris Lee, et al. (2014)
DevOps: AAROC Ansible LDAP role release.
Available:http://dx.doi.org/10.5281/zenodo.11914.
https://github.com/AAROC/DevOps/tree/v0.0.2
DOIDOI 10.5281/zenodo.1023310.5281/zenodo.10233

Sounds great, show me the details

Application porting, testing, integration and deployment workflow
Follow up
with the user
Fail Test
Tests Fail
Invoke Artifact
Tests Pass
Open Issue
on Github
Stage to
Testing Repo
Field Tests
Update Repo
Version
Lock CVMFS
repo
Field Tests Pass
Open Issue
on Github
Field Tests Fail
DeployArtifact
Update Repo
Version
Tag sites
Create
Build Script
Pull Request
Create Jenkins
Job
Build
if first time
Invoke Test
if build
has passed
Fail Build
Build Fails Build Passes
Follow up
with the user
Fail Test
Tests Fail
Invoke Artifact
Tests Pass
Open Issue
on Github
Stage to
Testing Repo
Update Repo
Version
The new pipeline

Public View

Authentication / Authorisation

Authenticated View

Does it work !?

Yes, it works.

Let's get very explicit

Debugging

cat <<MODULE_FILE
#%Module1.0
## $NAME modulefile
##
proc ModulesHelp { } {
    puts stderr "       This module does nothing but alert the user"
    puts stderr "       that the [moduleinfo name] module is not available"
}
preqreq("gsl","fftw/2.1.5","hdf5")
modulewhatis   "$NAME $VERSION."
setenv       GSL_VERSION       $VERSION
setenv       GSL_DIR           /apprepo/$::env(SITE)/$::env(OS)/$::env(ARCH)/$NAME/$VERSION
prependpath LD_LIBRARY_PATH   $::env(GSL_DIR)/lib
MODULE_FILE
) > modules/$VERSION
cat <<MODULE_FILE
#%Module1.0
## $NAME modulefile
##
proc ModulesHelp { } {
    puts stderr "       This module does nothing but alert the user"
    puts stderr "       that the [moduleinfo name] module is not available"
}
preqreq("gsl","fftw/2.1.5","hdf5")
modulewhatis   "$NAME $VERSION."
setenv       GSL_VERSION       $VERSION
setenv       GSL_DIR           /apprepo/$::env(SITE)/$::env(OS)/$::env(ARCH)/$NAME/$VERSION
prependpath LD_LIBRARY_PATH   $::env(GSL_DIR)/lib
MODULE_FILE
) > modules/$VERSION
Generic build script.
# GADGET requires HDF5 FFTW2 ZLIB and
openmpi
module add ci
module add fftw/2.1.5
module add hdf5
module add openmpi
module add gsl
# GADGET requires HDF5 FFTW2 ZLIB and
openmpi
module add ci
module add fftw/2.1.5
module add hdf5
module add openmpi
module add gsl
Set up the
environment
make install DESTDIR=$WORKSPACE/build
mkdir p $REPO_DIR
rm rf $REPO_DIR/*
tar cvzf $REPO_DIR/build.tar.gz C
$WORKSPACE/build apprepo
make install DESTDIR=$WORKSPACE/build
mkdir p $REPO_DIR
rm rf $REPO_DIR/*
tar cvzf $REPO_DIR/build.tar.gz C
$WORKSPACE/build apprepoActually build...
Create the artifact
Create the modulefile

There's gotta be a better way !
Literate/YAML/Templates...
---
author: Bruce Becker
repo: https://github.com/SAGridOps/SoftwareInstallation
name: GCC
target:
- sites:
- generic
- mpi
- mpi_infiniband
- OS:
- sl6
- u1404
versions:
- 4.6.1
category: compilers
dependencies:
- [ name: mpfr, version: 3.1.2 ]
url: http://gnu.org/path/to/gcc.tar.gz
build_script: build.sh

Last Bit: Delivery
Executable Artifacts
go here
Every site can
mount
the CVMFS
repo
Even your
laptop...

Sound Familiar ?
Executable Artifacts
go here
This will take a lot of
work, but there's a
huge community
behind the tools.
Deploy into linux
containers

Do you even remember the title of
this talk ?
EVERYTHING
=
CODE

What if we could automate
infrastructure execution too ?
●
We can and are, but there's no time to get into that right
now.
●
It's a jungle out there – but we know most of the animals.
●
Configuration and integration are the main issues.
●
Turn it into … a zoo ? (kinda)
●
See https://github.com/AAROC/DevOps
●
Yes, we need help

Summary
●
Making the best use of computational infrastructure comes
down to running applications
●
Maintaining and porting them is hard and tricky.
Communication is the hardest part.
●
We've built an automated porting system, which
●
will deliver functional, tested, relevant software to your site.
●
Can ease the communication blocks in collaborative work
●
Oh and it works :)
●
Come on in and help us build it.
We want an Open e-Science Commons

What can you do ?
● Join the orgs on Github
● Propose applications
● Fork, branch, pull request.
● Help develop the workflow
● Tell your friends

Thanks !
●
Join the organisations:
●
http://github.com/AAROC/
●
http://github.com/SAGridOps
●
Let's talk : http://aaroc.github.com/
●
https://ciari.slack.com/ (ask for an invite)
●
@TheSAGrid
●
@brusisceddu
●
fb.com/SAGrid

Continuous Integration and Delivery of Research Applications

Recommended

Recommended

More Related Content

Similar to Continuous Integration and Delivery of Research Applications

Similar to Continuous Integration and Delivery of Research Applications (20)

More from Bruce Becker

More from Bruce Becker (6)

Recently uploaded

Recently uploaded (20)

Continuous Integration and Delivery of Research Applications

Editor's Notes