Sergio Maffioletti from the University of Zurich presented this deck at the Switzerland HPC Conference. "This talk will present how the University of Zurich leverage an OpenStack-based cloud infrastructure to support a large variety of scientific usecases giving end-users the possibility of building their own dedicated research infrastructure. It will cover use cases spawning from large-scale data analysis to personalized pre/post processing services to customized data store solutions." Watch the video presentation: https://www.youtube.com/watch?v=w87gRzXu2KI See more talks in the Swiss Conference Video Gallery: http://insidehpc.com/2016-swiss-hpc-conference/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

1. S3IT: Service and Support for Science IT
OpenStack for academic
use-cases ?
Sergio Maffioletti
S3IT: Service and Support for ScienceIT,
University of Zurich
22.03.2016

2. What is S3IT and who am I
S3IT - Service and Support for ScienceIT:
http://www.s3it.uzh.ch.
Enabling research through Science IT competences.
S3IT is an academic solution provider (end-to-end).
Sergio Maffioletti
Senior Infrastructure and Application specialist at
University of Zurich.
UZH/S3IT OpenStack for academic use-cases 22.03.2016

3. Overall message
Build and adapt
infrastructure to use-cases
and not the other way round.
UZH/S3IT OpenStack for academic use-cases 22.03.2016

4. Let’s start with some
examples
UZH/S3IT OpenStack for academic use-cases 22.03.2016

5. Research needs to do more than just data
analysis.
Personalized server
Large-scale data analysis and simulations
Integrate solution
UZH/S3IT OpenStack for academic use-cases 22.03.2016

6. Research needs to do more than just data
analysis.
Personalized server
→ RStudio/R Shiny
→ Windows image analysis
Large-scale data analysis and simulations
Integrate solution
UZH/S3IT OpenStack for academic use-cases 22.03.2016

7. Research needs to do more than just data
analysis.
Personalized server
Large-scale data analysis and simulations
→ Hadoop/Spark
→ Own SLURM cluster
→ Matlab and toolboxes
Integrate solution
UZH/S3IT OpenStack for academic use-cases 22.03.2016

8. Research needs to do more than just data
analysis.
Personalized server
Large-scale data analysis and simulations
Integrate solution
→ from instrument to store, to processing to make
data accessible.
UZH/S3IT OpenStack for academic use-cases 22.03.2016

9. What is OpenStack ?
UZH/S3IT OpenStack for academic use-cases 22.03.2016

10. What is OpenStack and how we use it
→ Self-provisioning of Virtual Instances, Storage and
Network
→ Elastic allocation of resources on demand
→ Multi-tenancy
UZH/S3IT OpenStack for academic use-cases 22.03.2016

11. OpenStack project
OpenSource project
ask our sysadmin why this is important. . .
Foundation supported by more than 200 companies.
https://www.openstack.org/foundation/companies/
Network APIs
→ Amazon compute and storage cloud compliant
→ Bindings for many languages
UZH/S3IT OpenStack for academic use-cases 22.03.2016

12. Let’s implement some of the
examples
UZH/S3IT OpenStack for academic use-cases 22.03.2016

13. What approach ?
→ Provision, Configure and Orchestrate base
components: network, storage, compute.
→ Build on top of baseline infrastructure.
→ Customize and tailor to use-cases needs.
UZH/S3IT OpenStack for academic use-cases 22.03.2016

14. Personalized server: R Shiny
UZH/S3IT OpenStack for academic use-cases 22.03.2016

15. Personalized server: R Shiny
1. Create own instance
2. Automated customization (e.g. ansible)
3. Application-specific settings
4. Save base image for further use
UZH/S3IT OpenStack for academic use-cases 22.03.2016

16. R Shiny: Create own instance
Instances are virtual machines that run inside the cloud.
UZH/S3IT OpenStack for academic use-cases 22.03.2016

17. R Shiny: Automated customization (e.g.
ansible)
Ansible is a free-software configuration management
platform for configuring and managing computers.
UZH/S3IT OpenStack for academic use-cases 22.03.2016

18. R Shiny: Application-specific settings
Security groups are sets of IP filter rules that are applied to
an instance’s networking.
UZH/S3IT OpenStack for academic use-cases 22.03.2016

19. R Shiny: Save base image for further use
The OpenStack snapshot mechanism allows to create new
images from running instance.
UZH/S3IT OpenStack for academic use-cases 22.03.2016

20. Create your own batch cluster: SLURM
UZH/S3IT OpenStack for academic use-cases 22.03.2016

21. Create your own batch cluster: SLURM
1. Provision cluster instances (1 frontend + X
compute)
2. Create own data volume and attach it to frontend
3. Install and configure frontend and compute
4. Export via NFS data volume to all compute nodes
5. Start NFS and slurm services
UZH/S3IT OpenStack for academic use-cases 22.03.2016

22. Create your own batch cluster: SLURM
. . . or use elasticluster: http://gc3-uzh-ch.github.io/elasticluster/
https://youtu.be/gkXkcHDd588
UZH/S3IT OpenStack for academic use-cases 22.03.2016

23. Why flexibility: from our perspective
→ Reduce total time to get actual results
→ Allow to run on multiple infrastructures
(portability)
→ Automate (more research, less development for
end-user)
UZH/S3IT OpenStack for academic use-cases 22.03.2016

24. Why flexibility: from user perspective
Self-provisioning and Elasticity of resources
Customization and control of the environment
Network API
UZH/S3IT OpenStack for academic use-cases 22.03.2016

25. Why flexibility: from user perspective
Self-provisioning and Elasticity of resources
→ End-users can allocate and release resources
when needed.
Customization and control of the environment
Network API
UZH/S3IT OpenStack for academic use-cases 22.03.2016

26. Why flexibility: from user perspective
Self-provisioning and Elasticity of resources
Customization and control of the environment
→ End-users can tailor the research infrastructure
to their specific needs.
Network API
UZH/S3IT OpenStack for academic use-cases 22.03.2016

27. Why flexibility: from user perspective
Self-provisioning and Elasticity of resources
Customization and control of the environment
Network API
→ To programmatically create and control an own
research infrastructure.
→ Build services on top
UZH/S3IT OpenStack for academic use-cases 22.03.2016

28. Downsides
→ Resource allocation: policy is basically done by
support team.
→ Efficient use of resources: users need to take
responsibility.
→ Need to have strong support team to complement
the offer to end-users.
→ Ops costs comparable as for regular batch cluster
but know-how has to be developed.
→ Storage solution has to be planned separately and
mostly depends on use-cases.
UZH/S3IT OpenStack for academic use-cases 22.03.2016

29. Conclusions
→ Cloud is flexible: build and adapt infrastructure to
use-cases and not the other way round.
→ Users have more possibilities but also more
responsibilities.
→ Easier for end-users to think and plan their
research infrastructure.
UZH/S3IT OpenStack for academic use-cases 22.03.2016