OpenNebulaConf 2014 - Dynamic virtual private clusters with OpenNebula and SGE - Lykle Voort

•

0 likes•466 views

This document discusses using OpenNebula and SGE to create dynamic virtual private clusters in a cloud computing environment. It describes SURFsara's HPC infrastructure, including a cloud with 30 large compute nodes. A typical cluster configuration and job process are outlined. The document also explains how a queue monitor script works to dynamically add and remove virtual nodes from job queues in the cloud cluster based on demand, using OpenNebula and SGE. Future plans for the cloud include upgrading OpenNebula, adding more nodes and storage, and integrating GPUs.

Dynamic virtual private clusters with OpenNebula and SGE
lykle.voort@surfsara.nl
SURFsara

HPC Infrastructures at SURFsara
•Cartesius: 40000 cpu cores (Top-500 #45)
•Lisa: 9000 cpu cores
•Grid: (LHC, Life sciences)
•Hadoop
•Cloud (30 x 32 cores, 256GB)

HPC Cloud architecture
800TB
Shared
Storage
(NFS)
2×10Gbps
4
Service
Nodes
Virtualised
Services
30 large
compute
nodes
.
.
.
.
.
10 small
compute
nodes

Who was using our cloud in 2014?
Cell genetics 45%
Linguistics 10%
Medicine 6%
Economy 4%
Marketing 5%
Ecology 4%
Geography 2%
Civil engineering 7%
Physics 5%
Business 3%
Computer sciences 7%
Other 2%

Our cluster is old, we want a new cluster... in your cloud!

A typical cluster
•Lots of worker nodes
•Central user management (NIS, LDAP)
•Shared home file system
•Local disks for fast I/O
•A job scheduler (Torque, SGE, SLURM)
•Fixed size, bare metal

Queue monitor
•~400 lines of ruby code
•Runs within an EventMachine mainloop
•Uses qstat to monitor queues
•Uses qconf to add/remove nodes to/from queues
•Uses OCA to start/stop nodes

Adding nodes...
Start VM
Inspect queue
jobs
waiting?
machine
started?
tell scheduler
wait...
wait...
qstat...
ruby OCA
yes
no
yes
no
ssh.. .
qconf

Removing nodes...
tell scheduler
Inspect queue
nodes
idle?
wait...
shutdown VM
yes
no
ruby OCA
qconf
qstat...

Does it work?
•Yes, in principle...
•But...

Future of our cloud
•OpenNebula 4.x (January 2015)
•More compute nodes (February 2015)
•Ceph storage (February 2015)
•Local SSDs (February 2015)
•GPUs

Conclusions
•Integration with OCA/XML-RPC is possible and flexible
•Know your users and what they want (cattle? pets?)

`At the StampedeCon 2013 Big Data conference in St. Louis, Shrikanth Shankar, Head of Engineering at Qubole, presented Cloud-Friendly Hadoop and Hive. The cloud reduces the barrier to entry for many small and medium size enterprises into analytics. Hadoop and related frameworks like Hive, Oozie, Sqoop are becoming tools of choice for deriving insights from data. However, these frameworks were designed for in-house datacenters, which have different tradeoffs from a cloud environment, and making them run well in the cloud presents some challenges. In this talk, Shrikanth Shankar, Head of Engineering at Qubole, describes how these experiences taught us to extend Hadoop and Hive to exploit these new tradeoffs. Use cases will be presented that show how the challenges at large scale at Facebook are now making it extremely easy for a significantly smaller end user to leverage these technologies in the cloud.

10 Years of OpenStack at CERN - From 0 to 300k cores

Belmiro Moreira

CERN, the European Laboratory for Particle Physics, provides the infrastructure and resources to thousands of scientists all around the world to uncover the mysteries of the Universe. In the quest to build a private Cloud Infrastructure to support its users, CERN started early evaluating the OpenStack project, building several prototypes and engaging with the community. Finally, in 2013 CERN released its production Cloud Infrastructure using OpenStack. Since then we moved from a few hundred cores to a multi-cell deployment spread between different regions. After 7 years deploying and managing OpenStack in production at a large scale, we now look back and discuss the challenges of building a massive scale infrastructure from 0 to +300K cores. In this talk we will dive into the history, architecture, tools and technical decisions behind the CERN Cloud Infrastructure over the years.

ScyllaDB @ Apache BigData, may 2016

Tzach Livyatan

Scylla Summit 2019 Keynote - Avi Kivity

ScyllaDB

Apache Cassandra Lunch #52: Airflow and Cassandra for Cluster Management

Anant Corporation

OpenNebula Conf 2014 | Lightning talk: OpenNebula Puppet Module - Norman Mess...

NETWAYS

Apache Cassandra Lunch #70: Basics of Apache Cassandra

Anant Corporation

In Cassandra Lunch #70, we discuss the Basics of Apache Cassandra and setup a stand-alone Apache Cassandra. Accompanying Blog: https://blog.anant.us/cassandra-launch-70-basics-of-apache-cassandra Accompanying YouTube: https://youtu.be/o-yU0mi4nzc Sign Up For Our Newsletter: http://eepurl.com/grdMkn Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/ Cassandra.Link: https://cassandra.link/ Follow Us and Reach Us At: Anant: https://www.anant.us/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Cassandra.Lunch: https://github.com/Anant/Cassandra.Lunch Email: solutions@anant.us LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/ Join The Anant Team: https://www.careers.anant.us

Many people associate kubernetes with cloud solutions exclusively. I tend to disagree. In this presentation I will explain the rationale and success criteria of this project. I will also describe in details our key findings, tools that we used along the way, incident management process and the benefits analysis. The presentation will also cover all lessons learned as well as pitfalls and drawbacks of this solution. Major incidents will be explained along with the correctional procedures. Moreover this talk will also be focused on the cost effectiveness of the solution in question. No minikube demos, no raspberry pi clusters, no obvious basics. Just the real world application of the tech that works and generates revenue.

Learning to Scale OpenStack

Rainya Mosher

Time Series Data in a Time Series World

MapR Technologies

http://bit.ly/1ALVcwR – MapR Director of Architecture and Enterprise Strategy Jim Scott presented a session titled “Time Series Data in a Time Series World.” His session focused on working with time series data including single-value, geospatial and log time series data. By focusing on enterprise applications and the data center, OpenTSDB will be used as an example to explain some of the key time series core concepts including when to use different storage models. Things Expo | San Jose, California - November 2014

Tsunami alerting with Cassandra (From 0 to Cassandra on AWS in 30 days)

andrei.arion

An earthquake occurs in the Sea of Japan. A tsunami is likely to hit the coast. The population must be warned by SMS. A datacenter has been dammaged by the earthquake. Will the alerting system still work ? Building this simple alerting system is a great way to start with Cassandra, as we discovered teaching a bigdata hands-on class in a french university. What were the reasons that made a majority of students to choose Cassandra to implement a fast, resiliant and high availability bigdata system to be deployed on AWS ? What were the common pitfalls, the modelling alternatives and their performance impact ? Project description(in French): https://github.com/lesfurets/bigdata-paristech-project-2014

Moving from CellsV1 to CellsV2 at CERN

Belmiro Moreira

CERN, the European Organization for Nuclear Research, is running for several years a large OpenStack Cloud that helps thousands of scientists to analyze the data from the LHC. In 2012, early in the design phase of the CERN Cloud we decided to use Nova Cells to enable the infrastructure to scale to thousands of nodes. Now with more than 280K cores spread across 70 cells that are hosted in two data centres we were faced with the challenge to migrate to Nova Cells V2 required in the Pike release. In this presentation, we will describe how Nova Cells allowed CERN to scale to thousands of nodes, its advantages and how we mitigate the implementation issues of Nova Cells V1. Next, we will cover how we upgraded Nova from Newton with Cells V1 to Pike with Cells V2. We will explain the steps that we followed and the issues that we faced during the upgrade. Finally, we will report our experience with Cells V2 at scale, its caveats and how we are mitigate them. What can I expect to learn? This presentation describes how CERN migrated from Cells V1 to Cells V2 when upgraded from Newton to Pike release. You will learn the procedures followed by CERN in order to migrate Cells V1 to Cells V2 in a large production environment. The issues found during the upgrade and how we mitigate them will be discussed. Also, we will present how Cells V2 behaves in a large scale deployment with serveral thounsands nodes in 70 cells.

Hyperloglog Lightning Talk

Simon Prickett

Machine Learning & Data Science in the Age of the GPU: Smarter, Faster, Better

IDEAS - Int'l Data Engineering and Science Association

09 yong.luo-ceph in-ctrip

Yong Luo

Hpc to OpenStack: Our journey

Arif Ali

Introducing Scylla Open Source 4.0

ScyllaDB

Since its inception, Scylla has offered a compelling alternative to Apache Cassandra, providing better performance for a lower cost of ownership. With Scylla Open Source 4.0 we continue to extend our CQL interface features and capabilities and also now provide an open source alternative to DynamoDB, allowing you to run your workloads anywhere, on any cloud provider, or on premises. Join ScyllaDB co-founders, CTO Avi Kivity and CEO Dor Laor, for a look at the new features in Scylla Open Source 4.0, and architectural and cost comparisons with the coming Cassandra 4.0. Topics will include: Improved consistency with our new Lightweight Transactions Scylla Operator for Kubernetes How we stack up against Apache Cassandra 4.0 Our “run anywhere” DynamoDB alternative

Ceph Day Santa Clara: Ceph at DreamHost

Ceph Community

Scylla Summit 2022: Rakuten’s Catalog Platform Migration from Cassandra to Sc...

ScyllaDB

The RCP/Rakuten Catalog Platform has been growing at a brisk speed over the last couple of years. Our original backbone was Cassandra. However, as they continued their growth, they internally started realizing that it was not suitable for our next stage of growth. As such, they started looking into ScyllaDB as a better ROI solution as well as a much more stable backend. The migration itself was challenging since this has to be done for a production live data processing pipeline with minimal impact on customers. In this talk, Hitesh Shah, Engineering Manager at Rakuten USA will dive deeper into challenges and takeaways. To watch all of the recordings hosted during Scylla Summit 2022 visit our website here: https://www.scylladb.com/summit.

Need for Time series Database

Pramit Choudhary

[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...

PROIDEA

They promise that IoT (Internet of Things) will conquer the world. But what will tackle billions of bytes that flow into our servers every hour? First released in 2013, InfluxDB is used by eBay, Cisco, IBM and other big companies. It’s a production proven time-series storage. During this talk we're going to get acquainted with it and see how InfluxDB can help to solve your problems. We’ll see how to quickly install it on Amazon Web Services platform and how it scales. And for the dessert, we’re going to draw pretty Grafana graphs using InfluxDB data.

HaaS: HPCC Systems as a Service – BYOD to the Cloud Party

HPCC Systems

From the 2017 HPCC Systems Community Day: Amazon Web Services (AWS) is the premier IaaS provider. It leads the pack by offering more and better services at lower prices. Furthermore, AWS continuously improves and innovates to stay in front. There are numerous reasons to use an IaaS for HPCC Systems instead of dedicated hardware, especially if the workload does not execute 24/7. AWS has developed several features and tools for launching clusters. CloudFormation provides users a tool to make creating and managing an AWS resources much easier. Foremost it consists of a template (CFT) that defines resources required. The template is parameterizable and flexible so that a single CFT can launch an HPCC Systems cluster with an arbitrary number of nodes, various amount of memory per node, and other configuration options. Second, an Amazon Machine Image (AMI) contains the information needed to launch a compute node, with appropriate software, and configure it for a specific operation. We developed a CFT and an AMI for HPCC Systems. Additionally, we developed a reference architecture for HPCC Systems in AWS. It is a typical N+1 cluster, N worker nodes and one node (or mode) for cluster wide services such as Dali. The architecture also has storage (i.e., EBS volumes) and networking (i.e., VPN) resources. Significant effort was expended to determine the best set of resources for HPCC Systems clusters. Furthermore, we created a program to create and manage HPCC Systems clusters in AWS from the command line. This talk will present the tools we created. It also explains and justifies the reference architecture and many of the configuration options. Vince Freeh Associate Professor, North Carolina State University Vincent W. Freeh is an associate professor of computer science at North Carolina State University. He received his Ph.D. in 1996 from the University of Arizona. His research focus is high-performance system software, with emphasis on filesystems, parallel and distributed systems, power-aware computing, and storage systems. Prof Freeh teaches courses in the above research areas as well as in compilers. He has more than 55 referred publications in numerous computer science conferences and scientific journals. He received an NSF CAREER Award and several IBM Faculty Development Awards. He was a captain in the US Army Corps of Engineers before entering graduate school for his MS. Chin-Jung Hsu PhD Student, North Carolina State University Chin-Jung Hsu is a Ph.D. candidate in Computer Science at North Carolina State University. His primary research interests include distributed systems, storage systems, and performance optimization. He interned at NetApp and AT&T Research Lab, where he applied machine learning techniques to distributed storage systems for ensuring performance guarantees. Chin-Jung is currently working on how to efficiently run HPCC Systems applications on the public clouds such as AWS and Azure.

State of systemd @ Facebook

Davide Cavalca

Ceph de facto storage backend for OpenStack

eNovance

Running a DynamoDB-compatible Database on Managed Kubernetes Services

ScyllaDB

With the release of Alternator, Scylla’s DynamoDB-compatible API, you can now take your locked-in DynamoDB workloads and run them anywhere. Scylla provides a cost-effective open source alternative to Amazon’s DynamoDB, deployable wherever a user would want: on-premises, on other public clouds like Microsoft Azure or Google Cloud Platform, still on AWS (such as the high-density i3en instances) or as a fully managed DBaaS. In this session, we will cover: - Scylla Alternator: Scylla’s Amazon DynamoDB-compatible API - Scylla Operator: Running Scylla Alternator on Kubernetes - Demo Alternator - Demo and explain DynamoDB on GKE

Cloud Architect Alliance #15: Openstack

Microsoft

2018 03 25 system ml ai and openpower meetup

Ganesan Narayanasamy

What's hot

InfluxDb and Grafana fighting with data

Ivan Vaskevych

Introduction to Apache Cassandra

Intan Marselly

Kubernetes on bare metal after a year in production - Michal Ambroziewicz

PROIDEA

Learning to Scale OpenStack

Rainya Mosher

Time Series Data in a Time Series World

MapR Technologies

Tsunami alerting with Cassandra (From 0 to Cassandra on AWS in 30 days)

andrei.arion

Moving from CellsV1 to CellsV2 at CERN

Belmiro Moreira

Hyperloglog Lightning Talk

Simon Prickett

Machine Learning & Data Science in the Age of the GPU: Smarter, Faster, Better

IDEAS - Int'l Data Engineering and Science Association

09 yong.luo-ceph in-ctrip

Yong Luo

Hpc to OpenStack: Our journey

Arif Ali

Introducing Scylla Open Source 4.0

ScyllaDB

Ceph Day Santa Clara: Ceph at DreamHost

Ceph Community

Scylla Summit 2022: Rakuten’s Catalog Platform Migration from Cassandra to Sc...

ScyllaDB

Need for Time series Database

Pramit Choudhary

[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...

PROIDEA

HaaS: HPCC Systems as a Service – BYOD to the Cloud Party

HPCC Systems

State of systemd @ Facebook

Davide Cavalca

Ceph de facto storage backend for OpenStack

eNovance

Running a DynamoDB-compatible Database on Managed Kubernetes Services

ScyllaDB

What's hot (20)

InfluxDb and Grafana fighting with data

Introduction to Apache Cassandra

Kubernetes on bare metal after a year in production - Michal Ambroziewicz

Learning to Scale OpenStack

Time Series Data in a Time Series World

Tsunami alerting with Cassandra (From 0 to Cassandra on AWS in 30 days)

Moving from CellsV1 to CellsV2 at CERN

Hyperloglog Lightning Talk

Machine Learning & Data Science in the Age of the GPU: Smarter, Faster, Better

09 yong.luo-ceph in-ctrip

Hpc to OpenStack: Our journey

Introducing Scylla Open Source 4.0

Ceph Day Santa Clara: Ceph at DreamHost

Scylla Summit 2022: Rakuten’s Catalog Platform Migration from Cassandra to Sc...

Need for Time series Database

[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...

HaaS: HPCC Systems as a Service – BYOD to the Cloud Party

State of systemd @ Facebook

Ceph de facto storage backend for OpenStack

Running a DynamoDB-compatible Database on Managed Kubernetes Services

Similar to OpenNebulaConf 2014 - Dynamic virtual private clusters with OpenNebula and SGE - Lykle Voort

Cloud Architect Alliance #15: Openstack

Microsoft

2018 03 25 system ml ai and openpower meetup

Ganesan Narayanasamy

Sanger OpenStack presentation March 2017

Dave Holland

Building a GPU-enabled OpenStack Cloud for HPC - Blair Bethwaite, Monash Univ...

OpenStack

Audience Level Intermediate Synopsis M3 is the latest generation system of the MASSIVE project, an HPC facility specializing in characterization science (imaging and visualization). Using OpenStack as the compute provisioning layer, M3 is a hybrid HPC/cloud system, custom-integrated by Monash’s R@CMon Research Cloud team. Built to support Monash University’s next-gen high-throughput instrument processing requirements, M3 is half-half GPU-accelerated and CPU-only. We’ll discuss the design and tech used to build this innovative platform as well as detailing approaches and challenges to building GPU-enabled and HPC clouds. We’ll also discuss some of the software and processing pipelines that this system supports and highlight the importance of tuning for these workloads. Speaker Bio Blair Bethwaite: Blair has worked in distributed computing at Monash University for 10 years, with OpenStack for half of that. Having served as team lead, architect, administrator, user, researcher, and occasional hacker, Blair’s unique perspective as a science power-user, developer, and system architect has helped guide the evolution of the research computing engine central to Monash’s 21st Century Microscope. Lance Wilson: Lance is a mechanical engineer, who has been making tools to break things for the last 20 years. His career has moved through a number of engineering subdisciplines from manufacturing to bioengineering. Now he supports the national characterisation research community in Melbourne, Australia using OpenStack to create HPC systems solving problems too large for your laptop.

Climb Technical Overview

Arif Ali

PyData Boston 2013

Travis Oliphant

Devnexus 2018

Roy Russo

From the Archives: Future of Supercomputing at Altparty 2009

Olli-Pekka Lehto

NSCC Training Introductory Class

National Supercomputing Centre Singapore

Deep Dive Into the CERN Cloud Infrastructure - November, 2013

Belmiro Moreira

NSCC Training - Introductory Class

National Supercomputing Centre Singapore

CSCfi Computing Services 12/2014

Olli-Pekka Lehto

Webinar - DreamObjects/Ceph Case Study

Ceph Community

Ceph Day Melbourne - Scale and performance: Servicing the Fabric and the Work...

Ceph Community

Introduction to HPC & Supercomputing in AI

Tyrone Systems

HPC in the Cloud

Amazon Web Services

Those who out-compute can many times out-compete. The cloud gives you access to a massive amount of compute power when you need it. This talk will present an introduction to HPC in the cloud, including, the benefits of HPC in the cloud, how to get started, some tools to use, and how you can manage data. We will showcase several examples of HPC in the cloud by a number of public sector and commercial customers. Created by: Dr. Jeff Layton, Principal, Solutions Architect

ABCI: AI Bridging Cloud Infrastructure for Scalable AI/Big Data

Hitoshi Sato

D’une infrastructure de virtualisation scripté à un cloud privé OpenNebula

OpenNebulaConf 2014 - Dynamic virtual private clusters with OpenNebula and SGE - Lykle Voort

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to OpenNebulaConf 2014 - Dynamic virtual private clusters with OpenNebula and SGE - Lykle Voort

Similar to OpenNebulaConf 2014 - Dynamic virtual private clusters with OpenNebula and SGE - Lykle Voort (20)

More from OpenNebula Project

More from OpenNebula Project (20)

Recently uploaded

Recently uploaded (20)

OpenNebulaConf 2014 - Dynamic virtual private clusters with OpenNebula and SGE - Lykle Voort