Handling scientific workloads using opennebula, Roland den Hollander, VU
1. ‹#› Het begint met een idee
HANDLING SCIENTIFIC WORKLOADS
USING OPENNEBULA
2. Vrije Universiteit Amsterdam
“IT for Research” is a sub-department of the
IT organization of Vrije Universiteit Amsterdam
small team (~7 fte), incl. myself (Roland)
a miniature SURFsara
> housing / hosting of scientific equipment
> high performance computing clusters
> storage of large datasets
> consultancy and advice
2 VU IT / IT for Research
WHO AM I &
WHAT DO WE DO
photo by: Andy Tanenbaum
3. Vrije Universiteit Amsterdam
The existing university IT environment:
Focus on stability and availability
> DTAP-street
> enterprise hardware
> proprietary software
> licenses, support contracts
> mainly Windows
> stretched datacenter
Researchers need:
> low prices
> fast changes
> flexible use of resources
3 VU IT / IT for Research
OUR USE CASE
4. Vrije Universiteit Amsterdam
low budget + small team = commodity hardware and find
intelligent open source products
Linux + KVM was already in use
> need for live-migration, clones, snapshots
> need for selfservice & web interface for users and admins
> easy resizing / scaling of VM’s (cpu/mem/disk)
> high availability / resiliency
No loadbalancers, firewalls, RAID, SAS, support contracts
> simple sata disks
> if hardware fails, replace it with spare parts
> no licenses, contracts, proprietary things
> KISS and design for failure
4 VU IT / IT for Research
CHALLENGES & RESTRICTIONS
5. Vrije Universiteit Amsterdam
Hypervisors
> easy choice based on license fees
Cloudy products:
oVirt was too unstable at the time
> decided to drop it after 3 months
Openstack is huge. and changes a lot.
> promising, but still danger of vendor lock-in
OpenNebula - a working POC in ~2 weeks
> simple enough for us mere mortals, quick to deploy
and easy to maintain
5 VU IT / IT for Research
CANDIDATES WE LOOKED AT
7. Vrije Universiteit Amsterdam
Reverse proxy using nginx + SSL
LetsEncrypt certificates
3 networks available for VM’s
> internal network (campus only)
> external network (internet only)
> storage network (10 gbit internal network)
Ceph (for now only RBD images)
> very stable storage solution
> great performance
> self-repairing
7 VU IT / IT for Research
THINGS WE ADDED
8. Vrije Universiteit Amsterdam
use ansible (or any other tool) to automate changes to the
hypervisors
(ab)use ec2_public_key for oneadmin SSH key
control VNC traffic by editing /etc/hosts
8 VU IT / IT for Research
THINGS WE ADDED (2)
server01:
10.0.0.1 vnc-if
10.0.0.2
10.0.0.3
server02:
10.0.0.1
10.0.0.2 vnc-if
10.0.0.3
server03:
10.0.0.1
10.0.0.2
10.0.0.3 vnc-if
10. Vrije Universiteit Amsterdam
Popular usage:
- Workstation “in the cloud”, cross-uni cooperation, OpenVPN
- Nextcloud + 2-factor auth.
- Deploying large # VM’s for courses
- DNA mapping (memory intensive, short tasks)
10 VU IT / IT for Research
OPENNEBULA AND SCIENTIFIC WORKLOADS
11. Vrije Universiteit Amsterdam
Mass deployment for practical courses:
11 VU IT / IT for Research
SCIENTIFIC WORKLOADS (2)
dataset
VM Persistent
disk
Non-
Persistent
disk
VM
VM
VM
VM
VM
VM
13. Vrije Universiteit Amsterdam
adding Marketplace apps
> easy way to recycle our work, stimulate self-service
RAFT consensus algorithm (https://raft.github.io/)
> cheap & easy OpenNebula master replication
> no need for mysql clustering
Automating VM changes
> oneflow - autoscaling of VM’s (number of VM’s and/or VM resources)
> onegate - push/get information to VM’s
Introduce cloudbursting (Azure / SURFcumulus)
> prepare image on OpenNebula, mass-deploy on a public cloud
Upgrade Ceph and use Bluestore (Luminous v12)
> better write performance, lower latency (see tomorrow)
13 VU IT / IT for Research
SOON™
14. ‹#› Het begint met een idee
14 Het begint met een idee
- Don’t customize too much
Every ~year, a new version with extra
features is available.
- The inevitable administration problem
handing out resources is easy, getting them
back is not.
- Ease of deployment is met with lack of
interest
Security risk
- Internal processes don’t change as fast
as technology
14 VU IT / IT for Research
THINGS WE’VE LEARNED