Turning OpenStack Swift into a VM storage platformwim_provoost
OpenStack Swift is the Object Storage project within OpenStack. Alas, due to technical hurdles (eventual consistency, blocks <> objects, …) it is impossible to run Virtual Machines directly on Swift. You need a layer in between Swift and the hypervisor which can overcome these hurdles. This is where Open vStorage comes in.
Open vStorage is an open-source VM storage router. It is a software layer (called the VM Storage Router) in between Virtual Machines and storage backends. It allows to abstract the backend from the Virtual Machine and creates a uniform, single namespace across multiple hosts. These VM Storage Routers (VSRs) operate like a grid leveraging local flash memory or SSDs and any storage back-end (S3 compatible object store, (distributed) filesystem, NAS) to provide an extremely high performance and reliable storage system.
One of the supported Object Stores is OpenStack Swift. Open vStorage is the only solution to turn OpenStack Swift into block storage for Virtual Machines. Through a Cinder Plugin it allows to create and manage volumes directly on top of OpenStack Swift. Combining Open vStorage with OpenStack Swift allows to create a scale-out, performing, VM-centric storage platform which neatly integrates with OpenStack.
Turning object storage into vm storagewim_provoost
Object Storage is today the standard to build scale-out storage. But due to technical hurdles it is impossible to run Virtual Machines directly from an Object Store. Open vStorage is the layer between the hypervisor and Object Store and turns the Object Store into a high performance, distributed, VM-centric storage platform.
Turning OpenStack Swift into a VM storage platformwim_provoost
OpenStack Swift is the Object Storage project within OpenStack. Alas, due to technical hurdles (eventual consistency, blocks <> objects, …) it is impossible to run Virtual Machines directly on Swift. You need a layer in between Swift and the hypervisor which can overcome these hurdles. This is where Open vStorage comes in.
Open vStorage is an open-source VM storage router. It is a software layer (called the VM Storage Router) in between Virtual Machines and storage backends. It allows to abstract the backend from the Virtual Machine and creates a uniform, single namespace across multiple hosts. These VM Storage Routers (VSRs) operate like a grid leveraging local flash memory or SSDs and any storage back-end (S3 compatible object store, (distributed) filesystem, NAS) to provide an extremely high performance and reliable storage system.
One of the supported Object Stores is OpenStack Swift. Open vStorage is the only solution to turn OpenStack Swift into block storage for Virtual Machines. Through a Cinder Plugin it allows to create and manage volumes directly on top of OpenStack Swift. Combining Open vStorage with OpenStack Swift allows to create a scale-out, performing, VM-centric storage platform which neatly integrates with OpenStack.
Turning object storage into vm storagewim_provoost
Object Storage is today the standard to build scale-out storage. But due to technical hurdles it is impossible to run Virtual Machines directly from an Object Store. Open vStorage is the layer between the hypervisor and Object Store and turns the Object Store into a high performance, distributed, VM-centric storage platform.
For this upcoming meetup, we welcome Patrick Eaton PhD, Systems Architect at Stackdriver, and Joey Imbasciano, Cloud Platform Engineer at Stackdriver.
What You'll Learn At This Meetup:
• Why Stackdriver chose Cassandra over other DB offerings
• Stackdriver's data pipeline that runs into Cassandra
• Operating Cassandra Running on AWS
• Stackdriver's approach to disaster recovery
Patrick and Joey will be presenting their use of Apache Cassandra at Stackdriver, some lesson's learned, technical tips and a Q&A to end the evening.
OpenStack and Ceph case study at the University of AlabamaKamesh Pemmaraju
The University of Alabama at Birmingham gives scientists and researchers a massive, on-demand, virtual storage cloud using OpenStack and Ceph for less than $0.41 per gigabyte. This is a session at the OpenStack summit given by Kamesh Pemmaraju at Dell and John Paul at University of Alabama. This will detail how the university IT staff deployed a private storage cloud infrastructure using the Dell OpenStack cloud solution with Dell servers, storage, networking and OpenStack, and Inktank Ceph. After assessing a number of traditional storage scenarios, the University partnered with Dell and Inktank to architect a centralized cloud storage platform that was capable of scaling seamlessly and rapidly, was cost-effective, and that could leverage a single hardware infrastructure for the OpenStack compute and storage environment.
For this upcoming meetup, we welcome Patrick Eaton PhD, Systems Architect at Stackdriver, and Joey Imbasciano, Cloud Platform Engineer at Stackdriver.
What You'll Learn At This Meetup:
• Why Stackdriver chose Cassandra over other DB offerings
• Stackdriver's data pipeline that runs into Cassandra
• Operating Cassandra Running on AWS
• Stackdriver's approach to disaster recovery
Patrick and Joey will be presenting their use of Apache Cassandra at Stackdriver, some lesson's learned, technical tips and a Q&A to end the evening.
OpenStack and Ceph case study at the University of AlabamaKamesh Pemmaraju
The University of Alabama at Birmingham gives scientists and researchers a massive, on-demand, virtual storage cloud using OpenStack and Ceph for less than $0.41 per gigabyte. This is a session at the OpenStack summit given by Kamesh Pemmaraju at Dell and John Paul at University of Alabama. This will detail how the university IT staff deployed a private storage cloud infrastructure using the Dell OpenStack cloud solution with Dell servers, storage, networking and OpenStack, and Inktank Ceph. After assessing a number of traditional storage scenarios, the University partnered with Dell and Inktank to architect a centralized cloud storage platform that was capable of scaling seamlessly and rapidly, was cost-effective, and that could leverage a single hardware infrastructure for the OpenStack compute and storage environment.
Inter connect2016 yss1841-cloud-storage-options-v4Tony Pearson
This session will cover private and public cloud storage options, including flash, disk and tape, to address the different types of cloud storage requirements. It will also explain the use of Active File Management for local space management and global access to files, and support for file-and-sync.
The Pendulum Swings Back - Understanding Converged and Hyperconverged Integrated Systems, presented Oct 17, 2017 at IBM Systems Technical University, New Orleans LA
VMworld 2013: Extreme Performance Series: Storage in a Flash VMworld
VMworld 2013
Sankaran Sivathanu, VMware
Mark Achtemichuk, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
Whats new in Microsoft Windows Server 2016 Clustering and StorageJohn Moran
In this webinar we will learn what the High Availability & Storage team in Microsoft has cooked up for us in Windows Server 2016, which is being launched at Microsoft Ignite at the end of September.
There’s lots of new stuff in this release, including better high availability for Hyper-V, greater control over resource utilization, improved fault tolerance of transient events, newer design options for stretch or multi-site clusters, a whole new way of doing software defined storage with SATA and NVMe drives, built-in block-level storage replication, and hyper-convergence without having to break the bank.
Percona Live 4/14/15: Leveraging open stack cinder for peak application perfo...Tesora
In this session, speakers Amrith Kumar (Tesora), Steven Walchek (SolidFire), and Chris Merz (SolidFire) discuss Cinder, the OpenStack block storage service, and OpenStack Trove.
Leveraging OpenStack Cinder for Peak Application PerformanceNetApp
Deploying performance sensitive, database-driven applications in OpenStack can be tenuous if you are unsure how to utilize the Cinder API to get the most out of your OpenStack block storage.
This presentation:
Introduces Cinder, the OpenStack block storage service
Talks about the unique attributes of performance-sensitive applications and what this means in OpenStack
Walks you through how to use Cinder volume types and extra specs to guarantee performance to your various cloud workloads
Discusses OpenStack Trove and what it means for running database as a service in your OpenStack cloud
Container Attached Storage (CAS) with OpenEBS - Berlin Kubernetes Meetup - Ma...OpenEBS
The OpenEBS project has taken a different approach to storage when it comes to containers. Instead of using existing storage systems and making them work with containers; what if you were to redesign something from scratch using the same paradigms used in the container world? This resulted in the effort of containerizing the storage controller. Also, as applications that consume storage are changing over, do we need a scale-out distributed storage systems?
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
GridMate - End to end testing is a critical piece to ensure quality and avoid...ThomasParaiso2
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
2. CloudFounders
vRUN
Converged infrastructure that
combines the benefits of the
hyperconverged approach yet
offers independent compute
and storage scaling.
Open vStorage
Core Storage Technology
FlexCloud
A hosted private cloud based
on the vRun technology
available at multiple data
centers world-wide.
A product by CloudFounders
3. 2 Types of Storage
Block Storage:
•EMC, Netapp, ...
•Virtual Machines
•High perfomance, low latency
•Small capacity, typically fixed size
•Expensive
•Zero-copy snapshots, linked clones
•$/IOPS
Object Storage:
•Swift, Cleversafe, ...
•Unstructured data
•Low performance, high latency
•Large capacity, scalable
•Inexpensive, commodity hardware
•No high-end datamanagement features
•$/GB
What is needed is a technology which offers Virtual Machines the
performance and high-end features of a SAN but also the benefits
of the low cost and scale-out capabilities of object storage!
What is needed is a technology which offers Virtual Machines the
performance and high-end features of a SAN but also the benefits
of the low cost and scale-out capabilities of object storage!
4. What is Open vStorage
Open vStorage is an open-source software solution which creates a VM-centric, reliable,
scale out and high performance storage layer for OpenStack Virtual Machines on top of
object storage or a pool of Kinetic drives.
6. The architecture
OpenStack
Scale-outVM VM
VM VM
SSDSSD
SSDSSD
Open
vStorage
Open
vStorage
OpenStack
VM VM
VM VM
SSDSSD
SSDSSD
Open
vStorage
Open
vStorage
OpenStack
VM VM
VM VM
SSDSSD
SSDSSD
Open
vStorage
Open
vStorage
Unified Namespace
S3 compatible Object Storage
or a pool of Kinectic Drives
S3 compatible Object Storage
or a pool of Kinectic Drives
Tier 1 - Location Based
•Read/Write cache on SSD
•Block based storage
•Thin provisioning
•VM Centric
•Distributed Transaction Log
Tier 1 - Location Based
•Read/Write cache on SSD
•Block based storage
•Thin provisioning
•VM Centric
•Distributed Transaction Log
Tier 2 -Time Based
•Zero Copy Snapshot
•Zero Copy Cloning
•Continuous data protection
•Redundant storage
•Scale-out
Tier 2 -Time Based
•Zero Copy Snapshot
•Zero Copy Cloning
•Continuous data protection
•Redundant storage
•Scale-out
7. Optimized storage architecture
Powered by
Memory & SSD
Deduplicated Read Cache:
more effective use of Tier 1 storage.
Zero copy cloning
with linked clones
Thin Provisioning
Offload storage maintenance
tasks to the Tier 2
8. Unlimited scalability
Grow storage performance
by adding more SSDs
Grow storage capacity
by adding more disks
Asymmetric scalability
of CPU and storage
No bottlenecks
No dual controllers
9. Hyper Reliability
More reliable than raid5
Supports Live MigrationZero-shared architecture
Synchronized Distributed
Transaction Log
Unlimited snapshots,
longer retentions
10. Changes in Open vStorage 2 (End Q1 2015)
• Improved performance (x3) by tight integration with QEMU
– 50-70k iops per host
– Removes the NFS & FUSE performance loss
• Improved hardening against failure
– Seamless volume migration (no metadata rebuild)
– Limited impact of SSD failure
• Support for Seagate Kinetic drives as storage backend
– Encryption, compression, forward error correction
– Manage a pool of Kinetic drives as Tier2 storage
• Focus on OpenStack/KVM
11. Deduplicated Clustered Tier One (A pool of Flash)
Futher down the road ...
• Distributed Clustered Tier One
– Uses SSDs across the env. as 1 big shared, deduplicated Tier 1 read cache.
– Speed comparable with an All-Flash array: almost all VM I/O will be from flash.
– Scale storage performance by adding more SSDs.
– Limits impact of an SSD failure. Hot cache in case of Live Migration.
OpenStack
VM VM
VM VM
OpenStack
VM VM
VM VM
OpenStack
VM VM
VM VM
SSDSSD
SSDSSD
SSDSSD
SSDSSD
SSDSSD
SSDSSD
SSDSSD
SSDSSD
hashhash 4k block4k block
hashhash 4k block4k block
hashhash 4k block4k block
Scale-out
13. 2 Types of Storage
Block Storage:
•EMC, Netapp, ...
•Virtual Machines
•High perfomance, low latency
•Small capacity, typically fixed size
•Expensive
•Zero-copy snapshots, linked clones
•$/IOPS
Object Storage:
•Swift, Cleversafe, ...
•Unstructured data
•Low performance, high latency
•Large capacity, scalable
•Inexpensive, commodity hardware
•No high-end datamanagement features
•$/GB
What is needed is a technology which offers Virtual Machines the
performance and high-end features of a SAN but also the benefits
of the low cost and scale-out capabilities of object storage!
What is needed is a technology which offers Virtual Machines the
performance and high-end features of a SAN but also the benefits
of the low cost and scale-out capabilities of object storage!
14. OpenStack Swift: some highlights
• Designed to store unstructered data in a cost-effictive way
– Use low cost, large capacity SATA disks
– Increase capacity by adding more disk/servers when needed
– Increase performance by adding spindles/proxies
• High reliability by distributing content across disks
– 3 way replication
– Erasure coding (on the roadmap)
• Easy to manage (no knowledge needed about RAID or
volumes)
ProxyProxy ProxyProxy
Storage
Node
Storage
Node
Storage
Node
Storage
Node
Storage
Node
Storage
Node
15. Cinder: some highlights
• Cinder provides an infrastructure/API for managing volumes on OpenStack.
– Volume create, delete, list, show, attach, detach, extend
– Snapshot create, delete, list, show
– Backups create, restore, delete, list, show
– Manage volume types, quotas
– Migration
• By default Cinder uses local disks but plugins allow additional storage solutions to be
used:
– External appliances: EMC, Netapp, SolidFire
– Software solutions: GlusterFS, Ceph, …
16. Cinder with local disks has some problems ...
I S C S I
NovaCinder
ManagementN
ightmare!
17. A traditional OpenStack setup
Nova
Instance
Management
Nova
Instance
Management
Swift
Object Storage
Swift
Object Storage
Cinder
Block Storage
Cinder
Block Storage
Glance
Image store
Glance
Image store
VMVM
Provides
volume for
Provisions
Stores
image in
Stores backups in
Provides
image for
SAN,
NAS, ...
SAN,
NAS, ...
Provides
disk space
2 storage platforms?!2 storage platforms?!
18. “Swift under Cinder”?
• Eventual consistency (the CAP Theorem)
• Latency & performance
– VMs require low latency and high performance
– Object stores are developed to contain lots of data
(large disks, low performance)
– Additional latency as Object Store is on the Local LAN instead of attached to the host like DAS
• Different Management Paradigms
– Object Stores understand Objects <> Hypervisors understand blocks, files
19. Open vStorage & OpenStack
Nova
Instance
Management
Nova
Instance
Management
Swift
Object Storage
Swift
Object Storage
Cinder
Block Storage
Cinder
Block Storage
Glance
Image store
Glance
Image store
VMVM
Provides
volume for
Provisions
Stores
image in
Stores backups in
Provides
image for
Provides
disk spaceOpen vStorageOpen vStorageConverts Object Storage
into Block Storage
21. Get the software
• Open vStorage as open-source software is released under the Apache License, Version 2.0
– Free to use even in commercial products
– Open and free community help-forum : https://groups.google.com/forum/?hl=en#!
forum/open-vstorage
– You can contribute: https://bitbucket.org/openvstorage/
• Actively building a community
– Port of Open vStorage to CentOS
– Bug reporting and fixing
– Provide POC for new features
– ...
22. • To be released end Q1 2015
– Storage appliance + Open vStorage storage software
– Start package: 48TB storage + license for Open vStorage
(No restriction on amount of RAM, CPU and SSDs)
– Supported Open vStorage version
– Monitoring, support and maintenance included
– Low cost pricing
– ...
Open vStorage Based Storage Solution
23. • To be released end Q1 2015
– Converged OpenStack Solution based on Kinetic drives
– Starter package: 4 compute nodes, 48TB storage
– Supported Open vStorage version
– Supported OpenStack version
– Monitoring, support and maintenance included
– Low cost:
• 50% lower than EVO:RAIL
• 50% lower than Nutanix
Open vStorage Based Converged Solution
25. Open vStorage Summary
• +50.000 IOPS per Hypervisor
• Made for OpenStack Virtual Machines
• Unified Namespace
• Ultra Reliable
• Unlimited Snapshots
• Endless scalability for both capacity and storage
performance
• Lowest Management Cost In Market
S3 compatible
Object Based
Storage
S3 compatible
Object Based
Storage
Hypervisor
Open v StorageOpen v Storage
Hypervisor
Open v StorageOpen v Storage
Hypervisor
Open v StorageOpen v Storage
Hypervisor
Open vStorageOpen vStorage
Pool of
Kinetic Drives
Pool of
Kinetic Drives
30. Live Motion – In depth (Phase 2)
VSA 1 VSA 2 VSA 3
Arakoon – (config params, metadata, ...)
vDisk
1
vDisk
1
vDisk
2
vDisk
2
Internal
Bucket
vDisk
3
vDisk
3
VFS2 VFS3
xml
VOL
DRV
VOL
DRV
FILE
DRV
FILE
DRV
VM
VOL
DRV
Object Router
FILE
DRV
KVM1 KVM2 KVM3
VFS1
VMLive Motion
Object Router Handover Object Router
31. How does Open vStorage solve the problem
• Open vStorage is a middleware layer in between the hypervisor and the object store.
(Converts object storage into block storage)
– On the host: location based storage (block storage).
– On the backend: time based storage (ideal for objects stores).
– Open vStorage turns a volume into a single bucket.
• OpenStack Cinder Plugin for easy integration (snapshots, ...).
• Distributed file systems don’t work! Open vStorage is not a distributed file sysem!
– All hosts ‘think’ they see the same virtual file systems.
– Volume is ‘live’ on 1 host instead of all hosts.
– Only the virtual file system metadata is distributed.
• Caching inside the host fixes impedance mismatch between slow, high latency backend
and fast, low latency requirement of Virtual Machines.