TeideHPC is a High Performance Computing infrastructure, used for research and development tasks in a variety of areas like weather forecasting, astrophysics, CFD or bioinformatics. We are also involved in other fields of study, less related with R&D such as render or streaming services, and other with changing requirements, such the evaluation of pilot environments for open government/open data, research institutes or companies from the engineering sector. In those jobs we have found OpenNebula as a great solution with many capabilities but also with some limitations. The aim of these session is to show our experience with OpenNebula, from a technical viewpoint, presenting some topics on deployment/configuration with Cobbler/Chef, Infiniband virtualization with SR-IOV, Power management and Cloud bursting limitations. Author Biography Carlos Ignacio González Vila is a Software and System Engineer. Worked in the Research Support Computing Service in the same University for three years, doing a wide variety of tasks such as development of research applications for data analysis, project management or system administration of the clusters and supercomputers of the University. Since 2012 he is working as a High Performance Computing System Administrator in the Technological and Renewable Energies Institute (ITER) of Tenerife (Canary Islands), involved in the TeideHPC supercomputer proyect, a computing infrastructure with more than 1100 computing nodes and nearly 40 TB of RAM memory. His job includes storage management, ethernet and infiniband network, scientific application and cloud infrastructure.

1. OpenNebula implementation in
TeideHPC infrastructure
Carlos I. González Vila
OpenNebulaConf 2015 - Barcelona

2. ITER
Founded by the Cabildo Insular de Tenerife in 1990, the island's administrative
authority. Its objectives are to promote research activities and technological
development, especially those related with the use of the renewable energies.

3. ITER
ITER
MANAGEMENT
RENEWABLE
ENERGIES
Departamento de
Fotovoltaica
Departamento de Eólica
DISEMINATIONAREA
MAINTENANCE
Photovoltaics
Sustainable architecture
Wind Energy
INFORMATION
AND NEW
TECHNOLOGIES
Electronics
Informatics
Groundwater resources
Environmental quality
Volcanology
ENVIRONMENT
SCIENCE

4. TeideHPC

5. TeideHPC
● Weather forecasting
– Wind power generation
– Volcanic smoke dispersion prediction
– Solar power generation
● Bioinformatics
● Computational Fluid Dynamics
– Aerodynamics, aeroacoustics
– Industrial design

6. TeideHPC
● Astrophysics
– Solar composition
– Cosmological Parameters Estimation
– Analysis of solar spectro-polarimetric observations
● Computer sciences
● Streaming services
● Render
● Cloud computing

7. OpenNebula
● Implementation
● Study cases
– Infiniband virtualization
– Interoperability
● Use cases
– Multidatastore (lbox)
– Remote visualization
– Render farm

8. OpenNebula Implementation
● API
● Cobbler
● Chef
● NetApp Storage
● Monitoring
● Repository

9. OpenNebula Implementation
High Performance Computing Cloud computing
● Ethernet network configuration
● Infiniband configuration
● User auth & permissions
API

10. OpenNebula Implementation
● API
● Cobbler
● Chef
● NetApp Storage
● Monitoring
● Repository

11. OpenNebula Implementation
● API
● Cobbler
● Chef
● NetApp Storage
● Monitoring
● Repository

12. OpenNebula Implementation
● API
● Cobbler
● Chef
● NetApp Storage
● Monitoring
● Repository

13. OpenNebula Implementation
● API
● Cobbler
● Chef
● NetApp Storage
● Monitoring
● Repository

14. Study case – Infiniband
virtualization

15. Study case – Infiniband
virtualization
● OpenNebula Ecosystem
● KVM SR-IOV Driver
– Single root I/O Virtualization
Virtual
functions
PCIe device

16. Study case – Infiniband
virtualization
● OpenNebula Ecosystem
● KVM SR-IOV Driver
– Single root I/O Virtualization
Virtual
functions
PCIe device
# lspci
b0:00.0 Network controller: Mellanox Technologies MT27500 Family
[ConnectX-3]
b0:00.1 Network controller: Mellanox Technologies MT27500/MT27520 Family
[ConnectX-3/ConnectX-3 Pro Virtual Function]
b0:00.2 Network controller: Mellanox Technologies MT27500/MT27520 Family
[ConnectX-3/ConnectX-3 Pro Virtual Function]
● Enable option in BIOS
● Enable IO virtualization in kernel boot line
● Define number of Virtual Functions in kernel module load
● Burn SR-IOV capable firmware in the HCA (OEM)
● Upgrade OFED version (2.3-2.0.5-rhel6.6)

17. Study case – Infiniband
virtualization
● OpenNebula Ecosystem
● KVM SR-IOV Driver
● Great A'Tuin
– GPU devices support was funded by SURFsara
OpenNebula 4.14 Release notes... Support for GPU consumables,
giving the ability to give exlcusive PCI passthrough access to VMs to
GPU cards, for HPC computing.

18. Study case – Interoperability
limitations

19. Study case – Interoperability
limitations
● 2 research centers
– Share resources
● Conectivity
● Federation
● Cloud bursting

20. Study case – Interoperability
limitations
● 2 research centers
– Share resources
● Conectivity
● Federation
● Cloud bursting
● End-to-end connection
– L2-L3
– Jumbo frames
● 1 lambda - 10 Gbps
● Ping < 40 ms
● VPN IPsec
● +600Mbps between 1Gbps
hosts

21. Study case – Interoperability
limitations
● 2 research centers
– Share resources
● Conectivity
● Federation
● Cloud bursting
Tightly coupled

22. Study case – Interoperability
limitations
● 2 research centers
– Share resources
● Conectivity
● Federation
● Cloud bursting

23. Study case – Interoperability
limitations
● 2 research centers
– Share resources
● Conectivity
● Federation
● Cloud bursting
OpenNebula 4.8 docs... “The remote provider could be a
commercial Cloud service, such as Amazon EC2, IBM
SoftLayer or Microsoft Azure, or a partner infrastructure
running a different OpenNebula instance”

24. Use case – Multiple datastore

25. Use case – Multiple datastore
● Host load balancing
required
● Migrate / live migrate
● Time of migration (live)
– VM RAM size
– Host bandwith (IB)
– Network

26. Use case – Multiple datastore
● Storage overloaded
● Delete-recreate & Deploy to new
datastore
● Take image snapshot (same DS)
& create template & deploy
– VM HDD size
● Manual migration:
– Stop VM + copy image to new DS
+ delete-recreate + deploy +
replace image

27. Use case – Remote
visualization
● Lack of rendering hardware
● Low performance with ssh -X (even with
compression)
● Spice
– Audio+video
– Low BW requirements
● QXL driver (variable desktop resolution)
– 1024 x 768 … 2560 x 1600

28. Use case – Remote
visualization

29. Use case – Remote
visualization

30. Capture the Flag
● 404 VMs
● Connected to client private
infrastructure through
IPsec tunnel
● 20,8 TB HDD
● 12,8 TB RAM
● 12800 cores
● 100 KW

31. Thank you
teidehpc@iter.es
@teidehpc
teideHPC

32. References
● http://blog.scottlowe.org/2009/12/02/what-is-sr-iov/
● http://docs.opennebula.org/4.12/advanced_administration/data_center_federation/introf.html
● http://docs.opennebula.org/4.14/release_notes/release_notes/whats_new.html#opennebula-
core