OpenNebulaConf2017EU: Growing into the Petabytes for Fun and Profit by Michal Bielicki, Skytechnology

•

0 likes•273 views

Scale your OpenNebula into the Petabytes with LizardFS. Let us show you how to get from a small hyperconverged setup to a Petabyte cloud system utilising LizardFS with very little effort. YouTube: https://youtu.be/T-6GMwjgQjs

Technology

Scale out to petabytes with
LizardFS
Sixth OpenNebula Conference Madrid, October 2017

Think LizardFS ...
●  Just add chunkservers …
●  HW can be anything if used with
replication goals
o  Use your old spare or spare systems as
chunkservers
o  You can even use them temporary for a start and
move to faster systems later
●  Use different chunkserver groups for
different performance requirements …
o  Old stuff for archives
o  Hyperfast new rocketspeed for performance tier

And more ...
●  Multi Datacenter Support
●  NFS Support
●  Erasure Coding
●  Horizontaly and verticaly growable
and shrinkable
●  Transparent trash
●  Many more community and
commercial features

Special thanks go to our Partners that provided
us with the means for this demo:

Q&A
Do you have any questions ?
Regarding how things work ?
Regarding implementation ?
Why we did it ?
Other ?

Thank you
Michal Bielicki
m.bielicki@lizardfs.org
http://www.lizardfs.org/

We’re moving into a world of open cloud — where each organization can find the right cloud for its unique needs. A single cloud management platform can not be all things to all people, there will be a cloud space with several offerings focused on different environments and/or industries. The OpenNebula commitment to the open cloud flows directly out of its mission — to become the simplest cloud enabling platform — and its purpose — to bring simplicity to the private and hybrid enterprise cloud. OpenNebula exists to help companies build simple, cost-effective, reliable, open enterprise clouds on existing IT infrastructure. The OpenNebula Conference will be a great opportunity to remind our vision, vision and commitment, to look back at how the project has grown in the last 8 years, and to give a peek at what to expect from the project in the near future. YouTube: https://youtu.be/evzy5bLwDSM

OpenNebulaConf2017EU: Enabling Dev and Infra teams by Lodewijk De Schuyter,De...

OpenNebula Project

At the departement of environment and spatial planning we started 2 projects. The first was to replace our vmware based hosting environment with an open, hardware-vendor neutral, hypervisor environment. The second project’s goal was to enable our dev-teams more. This is the story of the second project. What we built and how it works using opennebula and ceph and our existing tooling. At the time of writing of this abstract, our opennebula environment is used by 4 dev-teams (almost 30 developers) and an infra team, hosting 700 virtual servers and counting. We are executing 300 deploys (as part of the development cycle) per week and counting … I will be talking about the setup we realized, the choices we made and the deployment tool we ended up with, integrating the toolset we already used. I.e. svn, ansible, opennebula, f5, jfrog, ubuntu/centos, zabbix, bareos, barman, opennebula, … YouTube: https://youtu.be/OEftbpJ_lSY

OpenNebulaConf2017EU: Transforming an Old Supercomputer into a Cloud Platform...

OpenNebula Project

Currently, typical supercomputers have an expected useful life of 3 or 4 years. One way of another, after this time period, infrastructure is typically replaced or upgraded to face the increasing resource demand by users and companies. This always gives rise to the same question, namely, what should be done with the old hardware if it was replaced? Possible solutions come in the form of decommissioning, splitting up and using it for spare parts, or donating it, but in several cases, the hardware can still provide value when used for different tasks. In this talk we will describe how we have converted the old Tier1 Flemish supercomputer https://www.ugent.be/hpc/en/infrastructure/tier1 into a cloud platform using OpenNebula. During this conversion process, we faced several technical challenges. The first and foremost of these was how to recycle hardware that was designed for use in a classical HPC environment to use in a private cloud. We will describe which steps were taken to address isolating VM traffic through the existing InfiniBand interconnect using the VXLAN network technology. We will also address how we managed mapping our internal (university) and external (industry) users using the OpenNebula “remote” authentication plugin. Finally, we will discuss how we used the InfiniBand interconnect to share the Ceph storage backend and VM traffic in a secure manner. After a testbed phase, in which only pilot users are given access and provide feedback, the new UGent HPC Cloud platform called “Grimer” will be available in production. YouTube: https://youtu.be/jHchktxIZnM

OpenNebulaConf2017EU: Testing Saltstack states with OpenNebula by Perry Manuk...

OpenNebula Project

OpenNebulaConf2017EU: Elastic Clusters for Data Analysis by Carlos de Alfonso...

OpenNebula Project

OpenNebulaConf2017EU: IPP Cloud by Jimmy Goffaux, IPPON

OpenNebula Project

OpenNebulaConf2017EU: FairShare Scheduling by Valentina Zaccolo, INDIGO

OpenNebula Project

CEPH DAY BERLIN - WELCOME

Ceph Community

This talk shows how to setup a virtual infrastructure using OpenNebula as cloud management platform, SaltStack for configuration management and Foreman for bare-metal/ virtual host provisioning. You will see how to combine OpenNebula with bare-metal deployment on standard server hardware using non-shared storage in an environment without physical access to the hardware and no existing base infrastructure like DNS, NTP, DHCP, VPN or other. The infrastructure installation has been done automatically using public code and free Open Source software.

rOCCI – Providing Interoperability through OCCI 1.1 Support for OpenNebula

NETWAYS

OCCI (Open Cloud Computing Interface ) [1] is an open protocol for management tasks in the cloud environment focused on integration, portability and interoperability with a high degree of extensibility. It is designed to bridge differences between various cloud platforms (or cloud middleware) and provide common ground for users and developers alike. The rOCCI framework [2], originally developed by GWDG [3], was written to simplify the implementation of the OCCI 1.1 protocol in Ruby and later provided the base for a working client and server implementation targeting OpenNebula as its primary back-end cloud platform. The initial server-side implementation provided basic functionality and served as a proof of concept when it was adopted by the EGI Federated Cloud Task Force [4] and chosen to act as the designated VM management interface. This led to further funding from EGI-InSPIRE [5] and involvement of CESNET [6]. This talk aims to provide basic information about the OCCI protocol, introduce its implementation in rOCCI, describe and/or demonstrate some of the functionality provided by rOCCI client and rOCCI-server in concert with OpenNebula. It also briefly examines its use in the EGI FedCloud environment and explores the possibility of further integration with OpenNebula as a part of the ON ecosystem or even as an integral part of OpenNebula itself in the future. All this with interoperability in mind.

OpenNebula Conf | Lightning talk: Managing a Scientific Computing Facility wi...

NETWAYS

SC4 Hangout - Luigi Selmi, Transport pilot architecture

BigData_Europe

OpenNebula Conf 2014: Expanding OpenNebula´s support for Cloud Bursting - Emm...

NETWAYS

The platform currently runs several solutions for Earth Sciences Researchers: Developer Cloud Sandboxes for scalable scientific processors integration, Virtual Archives federating distributed data repositories, Data Challenges for Earth Observation contests, and Digital Marketplaces for reproducible scientific experiments. OpenNebula is also powering our Cloud development environment, that has been enhanced during the past year. Terradue’s Cloud development environment is both our laboratory to test the latest developments from OpenNebula, especially for the integration of our own OpenNebula extensions, and our Engineering team facility to provision servers supporting project-based software developments. OpenNebula provides the virtualization and management of hardware clusters, that we rent from commercial ‘bare-metal’ providers. We have recently further developed several specific drivers for Multi-Cloud bursting, in order to provision virtual machines over public commercial clouds. When their processor integration and validation phase concludes, our researcher users can seamlessly burst their applications at scale, leveraging OpenNebula drivers for on-demand processing tasks.

Ceph Day Santa Clara Welcome

Ceph Community

OpenNebulaconf2017EU: OpenNebula 5.4 and Beyond by Tino Vázquez and Ruben S. ...

OpenNebula Project

In this talk, Rubén and Tino will lay our the novelties (not all of them, there are many!) present in 5.4, ranging from core new functionality to the big changes in vCenter. Also, the roadmap for 5.6 and future versions would be laid out, as far as it is consolidated (it won't be closed yet, but nearly so). It would also be the perfect session for feature requests, so don't miss it! YouTube: https://youtu.be/Czzm2EimayY

Thierry carrez openly developing open infrastructure

OpenInfra Days Poland 2019

OpenNebula Conf: 2014 | Lightning talk: Managing Docker Containers with OpenN...

NETWAYS

OpenStack Journey in Tieto Elastic Cloud

Jakub Pavlik

BDE SC4 Hangout - Hajira Jabeen, general architecture

BigData_Europe

Into the cold - Object Storage in SWITCHengines

Simon Leinen

OpenNebula Conf 2014 | The rOCCI project - a year later - alias OpenNebula in...

NETWAYS

Netflix Data Benchmark @ HPTS 2017

Ioannis Papapanagiotou

Cloud database vendors tend to report performance numbers for the sweet spot or running in highly optimized hardware with specific workload parameters. Moreover, many of these systems are not tested under different failure scenarios that may appear in the public cloud. At Netflix, as a cloud-native enterprise, our focus is on high availability. We achieve high availability by deploying at multiple regions. Hence, our data store system performance is highly affected by our global deployment model, instance types and workload patterns. Hence, we were interested in a Cloud database benchmark tool that could be deployed in a loosely-coupled fashion, as a microservice, and with the ability to dynamically change the configurations parameters at run time. In this paper, we present Netflix Data Benchmark (NDBench). NDBench offers pluggable patterns and loads and support for different client APIs. It offers the ability to deploy, manage and monitor multiple instances from a single point. NDBench was designed to run for infinite time. This nature offered us with the ability to test long-running database maintenance jobs, test database systems under conditions that affect the performance, such as compactions, repairs etc., and also gives us a view on client side issues like memory leaks and heap pressure. We have been running NDBench for almost 3 years having validated multiple database versions, tested numerous NoSQL systems running on the Cloud, and tested new functionalities. NDBench is major component of our testing and validation pipelines.

Ceph, storage cluster to go exabyte and beyond

Alvaro Soto

Cncf storage-final-filip

Juraj Hantak

Ryan Betts [InfluxData] | InfluxDB Platform Performance | InfluxDays Virtual ...

InfluxData

SC5 Hangout2 pilot 1 description

BigData_Europe

Ceph Day New York: Ceph: one decade in

Ceph Community

DevOps Fest 2020. Даніель Яворович. Data pipelines: building an efficient ins...

DevOps_Fest

What's hot

OpenNebulaConf2017EU: Providing cloud and Managed Hosting Environment by Mich...

OpenNebula Project

OpenNebulaConf2017EU: Hyper converged infrastructure with OpenNebula and Ceph...

OpenNebula Project

OpenNebula Conf 2014 | Bootstrapping a virtual infrastructure using OpenNebul...

NETWAYS

rOCCI – Providing Interoperability through OCCI 1.1 Support for OpenNebula

NETWAYS

OpenNebula Conf | Lightning talk: Managing a Scientific Computing Facility wi...

NETWAYS

SC4 Hangout - Luigi Selmi, Transport pilot architecture

BigData_Europe

OpenNebula Conf 2014: Expanding OpenNebula´s support for Cloud Bursting - Emm...

NETWAYS

Ceph Day Santa Clara Welcome

Ceph Community

OpenNebulaconf2017EU: OpenNebula 5.4 and Beyond by Tino Vázquez and Ruben S. ...

OpenNebula Project

Thierry carrez openly developing open infrastructure

OpenInfra Days Poland 2019

OpenNebula Conf: 2014 | Lightning talk: Managing Docker Containers with OpenN...

NETWAYS

OpenStack Journey in Tieto Elastic Cloud

Jakub Pavlik

BDE SC4 Hangout - Hajira Jabeen, general architecture

BigData_Europe

Into the cold - Object Storage in SWITCHengines

Simon Leinen

OpenNebula Conf 2014 | The rOCCI project - a year later - alias OpenNebula in...

NETWAYS

Netflix Data Benchmark @ HPTS 2017

Ioannis Papapanagiotou

Ceph, storage cluster to go exabyte and beyond

Alvaro Soto

Cncf storage-final-filip

Juraj Hantak

Ryan Betts [InfluxData] | InfluxDB Platform Performance | InfluxDays Virtual ...

InfluxData

SC5 Hangout2 pilot 1 description

BigData_Europe

What's hot (20)

OpenNebulaConf2017EU: Providing cloud and Managed Hosting Environment by Mich...

OpenNebulaConf2017EU: Hyper converged infrastructure with OpenNebula and Ceph...

OpenNebula Conf 2014 | Bootstrapping a virtual infrastructure using OpenNebul...

rOCCI – Providing Interoperability through OCCI 1.1 Support for OpenNebula

OpenNebula Conf | Lightning talk: Managing a Scientific Computing Facility wi...

SC4 Hangout - Luigi Selmi, Transport pilot architecture

OpenNebula Conf 2014: Expanding OpenNebula´s support for Cloud Bursting - Emm...

Ceph Day Santa Clara Welcome

OpenNebulaconf2017EU: OpenNebula 5.4 and Beyond by Tino Vázquez and Ruben S. ...

Thierry carrez openly developing open infrastructure

OpenNebula Conf: 2014 | Lightning talk: Managing Docker Containers with OpenN...

OpenStack Journey in Tieto Elastic Cloud

BDE SC4 Hangout - Hajira Jabeen, general architecture

Into the cold - Object Storage in SWITCHengines

OpenNebula Conf 2014 | The rOCCI project - a year later - alias OpenNebula in...

Netflix Data Benchmark @ HPTS 2017

Ceph, storage cluster to go exabyte and beyond

Cncf storage-final-filip

Ryan Betts [InfluxData] | InfluxDB Platform Performance | InfluxDays Virtual ...

SC5 Hangout2 pilot 1 description

Similar to OpenNebulaConf2017EU: Growing into the Petabytes for Fun and Profit by Michal Bielicki, Skytechnology

Ceph Day New York: Ceph: one decade in

Ceph Community

DevOps Fest 2020. Даніель Яворович. Data pipelines: building an efficient ins...

DevOps_Fest

Ceph Day SF 2015 - Keynote

Ceph Community

Ceph: A decade in the making and still going strong

Patrick McGarry

Ceph Day Santa Clara: Keynote: Building Tomorrow's Ceph

Ceph Community

Ceph Day NYC: Building Tomorrow's Ceph

Ceph Community

London Ceph Day Keynote: Building Tomorrow's Ceph

Ceph Community

Introduction into Ceph storage for OpenStack

OpenStack_Online

Advanced deeplearning4j features

Adam Gibson

F19 slidedeck (OpenStack^H^H^H^Hhift, what the)

Gerard Braad

PyData Boston 2013

Travis Oliphant

Hopsworks at Google AI Huddle, Sunnyvale

Jim Dowling

Keynote at Converge 2019

Travis Oliphant

OpenStack Cinder Best Practices - Meet Up

Aaron Delp

20160401 guster-roadmap

Gluster.org

20160401 Gluster-roadmap

Gluster.org

20160401 guster-roadmap

Gluster.org

AIDevWorldApacheNiFi101

Timothy Spann

AIDevWorld 23 Apache NiFi 101 Introduction and Best Practices https://sched.co/1RoAO Timothy Spann, Cloudera, Principal Developer Advocate In this talk, we will walk step by step through Apache NiFi from the first load to first application. I will include slides, articles and examples to take away as a Quick Start to utilizing Apache NiFi in your real-time dataflows. I will help you get up and running locally on your laptop, Docker or in CDP Public Cloud. Wednesday November 1, 2023 12:00pm - 12:25pm PDT VIRTUAL AI DevWorld -- Main Stage https://app.hopin.com/events/api-world-2023-ai-devworld/stages Retail & E-Commerce AI (Industry AI Conference) Session Type OPEN TALK Track or Conference Retail & E-Commerce AI (Industry AI Conference), Industry AI Conference, VIRTUAL, Tensorflow & PyTorch & Open Source Frameworks (AI/ML Engineering Conference), AI/ML Engineering Conference, AI DevWorld In-Person/Virtual Virtual, Virtual Exclusive apache nifi Timothy Spann Cloudera Principal Developer Advocate for Data in Motion Tim Spann is the Principal Developer Advocate for Data in Motion @ Cloudera where he works with Apache Kafka, Apache Flink, Apache NiFi, Apache Iceberg, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Developer Advocate at StreamNative, Principal Field Engineer at Cloudera, a Senior Solutions Architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science. cloudera dataflow

Public PaaS Throwdown!

Ronak Mallik

The Future of GlusterFS and Gluster.org

John Mark Walker

Similar to OpenNebulaConf2017EU: Growing into the Petabytes for Fun and Profit by Michal Bielicki, Skytechnology (20)

Ceph Day New York: Ceph: one decade in

DevOps Fest 2020. Даніель Яворович. Data pipelines: building an efficient ins...

Ceph Day SF 2015 - Keynote

Ceph: A decade in the making and still going strong

Ceph Day Santa Clara: Keynote: Building Tomorrow's Ceph

Ceph Day NYC: Building Tomorrow's Ceph

London Ceph Day Keynote: Building Tomorrow's Ceph

Introduction into Ceph storage for OpenStack

Advanced deeplearning4j features

F19 slidedeck (OpenStack^H^H^H^Hhift, what the)

PyData Boston 2013

Hopsworks at Google AI Huddle, Sunnyvale

Keynote at Converge 2019

OpenStack Cinder Best Practices - Meet Up

20160401 guster-roadmap

20160401 Gluster-roadmap

20160401 guster-roadmap

AIDevWorldApacheNiFi101

Public PaaS Throwdown!

The Future of GlusterFS and Gluster.org

More from OpenNebula Project

OpenNebulaConf2019 - Welcome and Project Update - Ignacio M. Llorente, Rubén ...

OpenNebula Project

We've made our way into the world of open cloud — where each organization can find the right cloud for its unique needs. A single cloud management platform cannot be all things to all people. There will be a cloud space with several offerings focused on different environments and/or industries. The OpenNebula commitment to the open cloud is at the very base of its mission — to become the simplest cloud enabling platform — and its purpose — to bring simplicity to the private and hybrid enterprise cloud. OpenNebula exists to help companies build simple, cost-effective, reliable, open enterprise clouds on existing IT infrastructure. The OpenNebula Conference will be a great opportunity to communicate and share our vision and commitment, to look back at how the project has grown in the last 9 years, and to shed some insight into what to expect from the project in the near future.

OpenNebulaConf2019 - Building Virtual Environments for Security Analyses of C...

OpenNebula Project

Computer networks are undergoing a phenomenal growth, driven by the rapidly increasing number of nodes constituting the networks. At the same time, the number of security threats on Internet and intranet networks is constantly increasing, and the testing and experimentation of cyber defense solutions require the availability of separate, test environments that best reflect the complexity of a real system. Such environments support the deployment and monitoring of complex mission-driven network scenarios, and cyber security training activities, thus enabling enterprises to study cyber defense strategies and allowing security researchers to evaluate their algorithms at scale. The main objective is delivering to researchers and practitioners an overview of the technological means and the practical steps to setup a private cloud platform based on OpenNebula for the creation and management of virtual environments that support cyber-security activities of training and testing, as well as an overview of its possible applications in the cyber security domain. In particular: 1. We describe our infrastructure based on OpenNebula 2. We overview our application, sitting on top of OpenNebula, as well as the technological tools involved in the management of its lifecycle (e.g., Ansible) . 3. We show how the platform can support various examples of security research activities  [References] Building an emulation environment for cyber security analyses of complex networked systems, Tanasache, Florin Dragos and Sorella, Mara and Bonomi, Silvia and Rapone, Raniero and Meacci, Davide, ICDCN '19, ACM, 2019

OpenNebulaConf2019 - CORD and Edge computing with OpenNebula - Alfonso Aureli...

OpenNebula Project

OpenNebulaConf2019 - 6 years (+) OpenNebula - Lessons learned - Sebastian Man...

OpenNebula Project

OpenNebulaConf2019 - Performant and Resilient Storage the Open Source & Linux...

OpenNebula Project

OpenNebula users have a range of storage options available to them, including proprietary appliances, proprietary software and Open Source software projects. This session will present a fully Open Source approach, that tightly integrates with Linux, and makes full use of the mature building blocks within the Linux kernel (LVM, Software RAID, DM-crypt, NVMe-oF Target, DRBD, etc...), and delivers one of the highest performance open source storage stacks currently available. The core goal is to expose the improved performance of NVMe storage devices to VMs and containers. The solution covers both local NVMe drives and NVMe-oF. For interacting with NVMe-oF targets it supports the Swordfish-API and LVM & Linux’s software NVMe-oF target. The solution contains a storage addon for OpenNebula.

OpenNebulaConf2019 - Image Backups in OpenNebula - Momčilo Medić - ITAF

OpenNebula Project

OpenNebulaConf2019 - How We Use GOCA to Manage our OpenNebula Cloud - Jean-Ph...

OpenNebula Project

At Iguane Solutions, a lot of our "DevOps" tools are developed in Golang, and we have a good amount of experience in contributing to the Goca. I'll review just what contributions we make, as well as how we use Goca with different tools, on a daily basis, to manage and monitor our OpenNebula cloud. I will delve into the concept of Infrastructure as Code - deployment of VM instances on cloud, as well as, also address the metrics collection of deployed VMs. Finally, I will present how we can abstract VM management with automation tools thanks to GOCA.

OpenNebulaConf2019 - Crytek: A Video gaming Edge Implementation "on the shoul...

OpenNebula Project

Replacing vCloud with OpenNebula

OpenNebula Project

NTS: What We Do With OpenNebula - and Why We Do It

OpenNebula Project

OpenNebula from the Perspective of an ISP

OpenNebula Project

NTS CAPTAIN / OpenNebula at Julius Blum GmbH

OpenNebula Project

Performant and Resilient Storage: The Open Source & Linux Way

OpenNebula Project

NetApp Hybrid Cloud with OpenNebula

OpenNebula Project

NSX with OpenNebula - upcoming 5.10

OpenNebula Project

Security for Private Cloud Environments

OpenNebula Project

CheckPoint R80.30 Installation on OpenNebula

OpenNebula Project

DE-CIX: CloudConnectivity

OpenNebula Project

DDC Demo

OpenNebula Project

Cloud Disaggregation with OpenNebula

OpenNebula Project

More from OpenNebula Project (20)