The OpenEBS Hangout #4 was held on 22nd December 2017 at 11:00 AM (IST and PST) where a live demo of cMotion was shown . Storage policies of OpenEBS 0.5 were also explained
Redis Conf 2019--Container Attached Storage for RedisOpenEBS
Kubernetes and containerized applications allow development teams to iterate fast, deploy efficiently and operate at scale. Kubernetes allows you to orchestrate containers that are highly available. However, in the case of container reschedule, Kubernetes does not provide a great set of primitives to manage your persistent data along with your application containers. In this talk, we will present some of the challenges associated with managing persistent data in Kubernetes and how we can make day 2 operations easier to manage. We will talk about a couple of approaches to solving data persistence problems in multi-cloud environments. During the demos, we will showcase how we address data replication and data encryption challenges.
Container Attached Storage (CAS) with OpenEBS - Berlin Kubernetes Meetup - Ma...OpenEBS
The OpenEBS project has taken a different approach to storage when it comes to containers. Instead of using existing storage systems and making them work with containers; what if you were to redesign something from scratch using the same paradigms used in the container world? This resulted in the effort of containerizing the storage controller. Also, as applications that consume storage are changing over, do we need a scale-out distributed storage systems?
cStor is a resilient storage engine built with proven building blocks of storage components. The storage block layer is derived from the user space ZFS inherited from the proven OpenSolaris stack. The volumes can be accessed via iSCSI Target which is derived from Linux to BSD port. Both of these core components of cStor have been field tested at thousands of installations for many years.
Redis Conf 2019--Container Attached Storage for RedisOpenEBS
Kubernetes and containerized applications allow development teams to iterate fast, deploy efficiently and operate at scale. Kubernetes allows you to orchestrate containers that are highly available. However, in the case of container reschedule, Kubernetes does not provide a great set of primitives to manage your persistent data along with your application containers. In this talk, we will present some of the challenges associated with managing persistent data in Kubernetes and how we can make day 2 operations easier to manage. We will talk about a couple of approaches to solving data persistence problems in multi-cloud environments. During the demos, we will showcase how we address data replication and data encryption challenges.
Container Attached Storage (CAS) with OpenEBS - Berlin Kubernetes Meetup - Ma...OpenEBS
The OpenEBS project has taken a different approach to storage when it comes to containers. Instead of using existing storage systems and making them work with containers; what if you were to redesign something from scratch using the same paradigms used in the container world? This resulted in the effort of containerizing the storage controller. Also, as applications that consume storage are changing over, do we need a scale-out distributed storage systems?
cStor is a resilient storage engine built with proven building blocks of storage components. The storage block layer is derived from the user space ZFS inherited from the proven OpenSolaris stack. The volumes can be accessed via iSCSI Target which is derived from Linux to BSD port. Both of these core components of cStor have been field tested at thousands of installations for many years.
Red Hat Gluster Storage, Container Storage and CephFS PlansRed_Hat_Storage
At Red Hat Storage Day New York on 1/19/16, Red Hat's Sayan Saha took attendees through an overview of Red Hat Gluster Storage that included future plans for the product, Red Hat's plans for container storage, and the company's plans for CephFS.
In this session, we'll discuss new volume types in Red Hat Gluster Storage. We will talk about erasure codes and storage tiers, and how they can work together. Future directions will also be touched on, including rule based classifiers and data transformations.
You will learn about:
How erasure codes lower the cost of storage.
How to configure and manage an erasure coded volume.
How to tune Gluster and Linux to optimize erasure code performance.
Using erasure codes for archival workloads.
How to utilize an SSD inexpensively as a storage tier.
Gluster's erasure code and storage tiering design.
Red Hat Gluster Storage - Direction, Roadmap and Use-CasesRed_Hat_Storage
Red Hat Gluster Storage is open, software-defined storage that helps you manage big, unstructured, and semistructured data. This product is based on the open source project GlusterFS, a distributed scale-out file system technology, and focuses on file sharing, analytics, and hyper-converged use cases.
In this session, you will:
See real-life case studies about Red Hat Gluster Storage’s usage in production environments, including ideal workloads.
Learn about the Red Hat Gluster Storage roadmap, including innovations from the GlusterFS community pipeline.
Gain insights into how the product will be integrated with Red Hat Enterprise Virtualization (including hyperconvergence), Red Hat Satellite, and Red Hat Enterprise Linux OpenStack Platform.
"Data classification" is an umbrella term covering things: locality-aware data placement, SSD/disk or normal/deduplicated/erasure-coded data tiering, HSM, etc. They share most of the same infrastructure, and so are proposed (for now) as a single feature.
Webinar: OpenEBS - Still Free and now FASTEST Kubernetes storageMayaData Inc
Webinar Session - https://youtu.be/_5MfGMf8PG4
In this webinar, we share how the Container Attached Storage pattern makes performance tuning more tractable, by giving each workload its own storage system, thereby decreasing the variables needed to understand and tune performance.
We then introduce MayaStor, a breakthrough in the use of containers and Kubernetes as a data plane. MayaStor is the first containerized data engine available that delivers near the theoretical maximum performance of underlying systems. MayaStor performance scales with the underlying hardware and has been shown, for example, to deliver in excess of 10 million IOPS in a particular environment.
Red Hat Gluster Storage, Container Storage and CephFS PlansRed_Hat_Storage
At Red Hat Storage Day New York on 1/19/16, Red Hat's Sayan Saha took attendees through an overview of Red Hat Gluster Storage that included future plans for the product, Red Hat's plans for container storage, and the company's plans for CephFS.
In this session, we'll discuss new volume types in Red Hat Gluster Storage. We will talk about erasure codes and storage tiers, and how they can work together. Future directions will also be touched on, including rule based classifiers and data transformations.
You will learn about:
How erasure codes lower the cost of storage.
How to configure and manage an erasure coded volume.
How to tune Gluster and Linux to optimize erasure code performance.
Using erasure codes for archival workloads.
How to utilize an SSD inexpensively as a storage tier.
Gluster's erasure code and storage tiering design.
Red Hat Gluster Storage - Direction, Roadmap and Use-CasesRed_Hat_Storage
Red Hat Gluster Storage is open, software-defined storage that helps you manage big, unstructured, and semistructured data. This product is based on the open source project GlusterFS, a distributed scale-out file system technology, and focuses on file sharing, analytics, and hyper-converged use cases.
In this session, you will:
See real-life case studies about Red Hat Gluster Storage’s usage in production environments, including ideal workloads.
Learn about the Red Hat Gluster Storage roadmap, including innovations from the GlusterFS community pipeline.
Gain insights into how the product will be integrated with Red Hat Enterprise Virtualization (including hyperconvergence), Red Hat Satellite, and Red Hat Enterprise Linux OpenStack Platform.
"Data classification" is an umbrella term covering things: locality-aware data placement, SSD/disk or normal/deduplicated/erasure-coded data tiering, HSM, etc. They share most of the same infrastructure, and so are proposed (for now) as a single feature.
Webinar: OpenEBS - Still Free and now FASTEST Kubernetes storageMayaData Inc
Webinar Session - https://youtu.be/_5MfGMf8PG4
In this webinar, we share how the Container Attached Storage pattern makes performance tuning more tractable, by giving each workload its own storage system, thereby decreasing the variables needed to understand and tune performance.
We then introduce MayaStor, a breakthrough in the use of containers and Kubernetes as a data plane. MayaStor is the first containerized data engine available that delivers near the theoretical maximum performance of underlying systems. MayaStor performance scales with the underlying hardware and has been shown, for example, to deliver in excess of 10 million IOPS in a particular environment.
Free GitOps Workshop + Intro to Kubernetes & GitOpsWeaveworks
Follow along in this free workshop and experience GitOps!
AGENDA:
Welcome - Tamao Nakahara, Head of DX (Weaveworks)
Introduction to Kubernetes & GitOps - Mark Emeis, Principal Engineer (Weaveworks)
Weave Gitops Overview - Tamao Nakahara
Free Gitops Workshop - David Harris, Product Manager (Weaveworks)
If you're new to Kubernetes and GitOps, we'll give you a brief introduction to both and how GitOps is the natural evolution of Kubernetes.
Weave GitOps Core is a continuous delivery product to run apps in any Kubernetes. It is free and open source, and you can get started today!
https://www.weave.works/product/gitops-core
If you’re stuck, also come talk to us at our Slack channel! #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
Bandwidth: Use Cases for Elastic Cloud on Kubernetes Elasticsearch
Bandwidth has been an avid user of the Elastic Stack for aggregating their logs from its many data centers. Learn how Bandwidth uses Elastic Cloud on Kubernetes to help satisfy various use cases.
Introduction to Container Storage Interface (CSI)Idan Atias
Among the cool stuff we do at Silk, my colleagues and I develop the Silk CSI Plugin for customers who use our system as the storage layer for their Kubernetes workloads.
Before deep diving into the code and as part of my ramp-up on this subject I prepared some slides that cover some basic and important information on this topic.
These slides start by recapping some basic storage principals in containers and Kubernetes, continues with some more advanced use cases (including an "offline demo" of persisting Redis data on EBS volumes), and ends with a detailed information on the CSI solution itself.
IMHO, reviewing these slides can improve your understanding on this matter and can get you started implementing your own CSI plugin.
The main sources of information I used for preparing these slides are:
* Official CSI docs
* Kubernetes Storage Lingo 101 - Saad Ali, Google
* Container Storage Interface: Present and Future - Jie Yu, Mesosphere, Inc.
MayaData Datastax webinar - Operating Cassandra on Kubernetes with the help ...MayaData Inc
In this webinar experts from DataStax - the lead developer of Cassandra - and from MayaData - the lead developer of OpenEBS and LitmusChaos - will discuss and demonstrate ways to ensure the ease of use and resilience of Cassandra on Kubernetes.
Topics to be discussed and demonstrated include:
Provisioning underlying storage - how to make it consistent irrespective of the underlying hardware or cloud? Are there are ever reasons to have the storage replicate across nodes or is dynamic LocalPV the best choice in all cases?
Cass Operator - DataStax Kubernetes Operator for Apache Cassandra
Resilience - how to proactively assess the overall environment including the underlying Kubernetes with the help of Litmus
Implementing data and databases on K8s within the Dutch governmentDoKC
A small walkthrough of projects within the dutch government running Data(bases) on OpenShift. This talk shares success stories, provides a proven recipe to `get it done` and debunks some of the FUD.
About Sebastiaan:
I have always been a weird DBA, trying to combine Databases with out-of-the-box thinking and a DevOps mindset. Around 2016 I fell in love with both Postgres and Kubernetes, and I then committed my life to enabling Dutch organisations with running their Database workloads CloudNative.
Over the last few years I worked as a private contractor for 2 large government agencies doing exactly that, and I want to share my and others (success stories) hoping to enable and inspire Data on Kubernetes adoption.
Join this info-packed and hands-on workshop where we will cover:
Introduction to Kubernetes & GitOps talk:
We'll cover the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
* an opinionated repo structure to minimize decision fatigue
* disaster recovery using GitOps
* Helm charts example
* Multi-cluster example
* all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
For this info-packed and hands-on workshop we cover:
📍 Introduction to Kubernetes & GitOps talk:
We cover the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
📍 Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
- an opinionated repo structure to minimize decision fatigue
- disaster recovery using GitOps
- Helm charts example
- Multi-cluster example
- all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
Como creamos QuestDB Cloud, un SaaS basado en Kubernetes alrededor de QuestDB...javier ramirez
QuestDB es una base de datos open source de alto rendimiento. Mucha gente nos comentaba que les gustaría usarla como servicio, sin tener que gestionar las máquinas. Así que nos pusimos manos a la obra para desarrollar una solución que nos permitiese lanzar instancias de QuestDB con provisionado, monitorización, seguridad o actualizaciones totalmente gestionadas.
Unos cuantos clusters de Kubernetes más tarde, conseguimos lanzar nuestra oferta de QuestDB Cloud. Esta charla es la historia de cómo llegamos ahí. Hablaré de herramientas como Calico, Karpenter, CoreDNS, Telegraf, Prometheus, Loki o Grafana, pero también de retos como autenticación, facturación, multi-nube, o de a qué tienes que decir que no para poder sobrevivir en la nube.
DevOps Days Boston 2017: Real-world Kubernetes for DevOpsAmbassador Labs
DevOps Days Boston 2017
Microservices is an increasingly popular approach to building cloud-native applications. Dozens of new technologies that streamline adopting microservices development such as Docker, Kubernetes, and Envoy have been released over the past few years. But how do you actually use these technologies together to develop, deploy, and run microservices?
In this presentation, we’ll cover the nuances of deploying containerized applications on Kubernetes, including creating a Kubernetes manifest, debugging and logging, and how to build an automated continuous deployment pipeline. Then, we’ll do a brief tour of some of the advanced concepts related to microservices, including service mesh, canary deployments, resilience, and security.
Kubernetes Forum Seoul 2019: Re-architecting Data Platform with KubernetesSeungYong Oh
Session Video: https://youtu.be/7MPH1mknIxE
In this talk, we share Devsisters' journey of migrating its internal data platform including Spark to Kubernetes, with its benefits and issues.
데브시스터즈에서 데이터플랫폼 컴포넌트를 쿠버네티스로 옮기면서 얻은 장점들과 이슈들에 대해 공유합니다.
Conference session page:
- English: https://sched.co/WIRK
- Korean: https://sched.co/WYRc
Workday has built one of the largest OpenStack-based private clouds in the world, hosting a workload of over a million physical cores on over 16,000 compute nodes in 5 data centers for over ten years. However, there was a growing need for a newer, more maintainable deployment model that would closely follow the upstream community. We would like to share our new architecture and deployment approach as well as lessons learned from our experience.
We’ve converted many of our technologies in the process, from…
Migrating from Mitaka, to Victoria
Converting from OpenContrail, to pure L3 Calico with BGP on the host
Deploying with Chef, to deploying with Ansible
Building home-grown container images, to Kolla
Monitoring with Sensu and Wavefront, to Prometheus and Grafana
CI/CD in Jenkins, to Zuul
CentOS 7, to CentOS 8 Stream
We'll also talk about some internal tools we wrote that, while Workday-specific, may inspire you to see what value-add you can make for your customers.
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)Kevin Lynch
In this presentation I talk about our motivation to converting our microservices to run on Kubernetes. I discuss many of the technical challenges we encountered along the way, including networking issues, Java issues, monitoring and alerting, and managing all of our resources!
Kubernetes and Cloud Native Update Q4 2018CloudOps2005
This year’s final set of Kubernetes and Cloud Native meetups just took place. They kicked off in Kitchener-Waterloo on November 29th, and continued in Montreal December 3rd, Ottawa December 4th, Toronto December 5th, and Quebec December 6th. In preparation for the upcoming KubeCon and CloudNativeCon in Seattle, a wide range of open source solutions were discussed and, as always, beer and pizza provided. Ayrat Khayretdinov began each meetup with an update of Kubernetes and the Cloud Native landscape.
[WSO2Con EU 2018] Deploying Applications in K8S and DockerWSO2
Within the last four years container technologies have become very popular. A lot of companies and developers are now using containers to ship their applications. Docker provides an easy-to-use packaging model to bundle the application. However in many cases, a single container is not enough to run an application. It requires multiple containers, scaled into multiple host machines to become a production grade deployment. Kubernetes is an open source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. This presentation discusses best practices of deploying application in Docker and Kubernetes while discussing Docker and Kubernetes concepts.
Latest (storage IO) patterns for cloud-native applications OpenEBS
Applying micro service patterns to storage giving each workload its own Container Attached Storage (CAS) system. This puts the DevOps persona within full control of the storage requirements and brings data agility to k8s persistent workloads. We will go over the concept and the implementation of CAS, as well as its orchestration.
About the Talk:
Cloud native ecosystem is bringing a huge change in the way of DevOps in every cloud native organisation. Developers and operators in cloud native organisations are using tools and platforms like Kubernetes to achieve the agility promised by DevOps and microservices. The tools and best practices for stateless applications have been well established and the results can be seen in the agility of teams using these stateless applications. However, stateful applications pose new challenges to DevOps teams in achieving the agility as the best practices around persistent storage management are still emerging. In this talk, first we discuss the challenges faced by DevOps while dealing with persistent storage handling in stateful applications. Then we discuss the open source tools and best practices for DevOps teams to achieve data agility of cloud native applications.
Introduction to cStor replica - Contributors Meet 5th Oct 2018OpenEBS
cStor storage engine has separate container image files for storage controller and storage replica. Docker images for controller is at << https://hub.docker.com/r/openebs/cstor-controller/>> and for replica is at << https://hub.docker.com/r/openebs/cstor-pool/>>. cStor is a high performing storage engine built with proven building blocks of storage components. Access protocol iSCSI stack is a linux ported BSD based Multi-threaded iSCSI protocol stack originally developed at CloudByte. This iSCSI is field tested at thousands of installations for many years". The storage block layer is the DMU layer of user space ZFS inherited from the proven OpenSolaris stack. With these proven building blocks, cStor engine is highly reliable for storing and protecting enterprise data.To know more details click https://docs.openebs.io/docs/next/storageengine.html#cstor
Container Attached Storage (CAS) with OpenEBS - SDC 2018OpenEBS
Applying micro service patterns to storage giving each workload its own Container Attached Storage (CAS) system. This puts the DevOps persona within full control of the storage requirements and brings data agility to k8s persistent workloads. We will go over the concept and the implementation of CAS, as well as its orchestration.
The added features include in 0.7 includes
Management of local nodes
Many NoSQL and other users do not need the resilience features of OpenEBS however they do want management of these nodes. OpenEBS users can now designate local node via their storage classes and have the benefits of metrics and fault management.
Node Disk Management support (NDM). One capability that has been added by MayaData to Kubernetes itself is improvements in the management of storage media in local nodes. OpenEBS 0.7 now uses NDM to improve the resilience and flexibility of Kubernetes for storage.
Data operations automation
As OpenEBS was adopted users increasingly encountered a number of storage and data management tasks such as data clean up, data back-up, copy data operations, auto-scaling and cross availability zone deployment specification and much more. All of these use cases and others have been made simpler to deliver in OpenEBS, to test and validate via Litmus integration in MayaOnline, and to observe and operate in MayaOnline.
Resilience and supportability
The fundamental task of any storage is to keep the data safe. With that in mind, OpenEBS adds resilience and supportability capabilities in each release, such as improved retry logic, improved resilience to network outages, cross availability zone recovery, initial snapshot-based replication, and backup options, and ongoing integrations with emerging Kubernetes capabilities to further increase control and availability.
A new user interface for MayaOnline
MayaOnline has been redesigned to be more intuitive for use in understanding, controlling and troubleshooting stateful environments on Kubernetes.
Thoughts on heptio's ark - Contributors Meet 21st Sept 2018OpenEBS
Create a OpenEBS ARK Plugin that will implement the Block-Store API exposed by ARK
Backup Operation
ARK will invoke the Plugin-Snapshot (Backup) method.
Plugin: will call maya-apiserver backup api on a given volume
Maya-apiserver backup will call volumes (jiva/cstor) backup api
(jiva/cstor) volume controller will take a snapshot, and pass the request to one of the replica’s to push snapshot data to remote backup location (say S3 compatible -- as passed via the ark plugin or a custom backup location on mayaonline or may be a nfs server that openebs supports ). The code to actually push the data to backup location can make use of restic. We are putting it at the jiva/cstor for getting access to snapshot/incr snapshot data.
Restore Operation
ARK will invoke the Plugin-VolumeFromSnapshot (Restore) method
Plugin will invoke maya-apiserver to create a new PV/PVC and restore the data from backup.
ARK will launch the application with the PV/PVC.
When one of the zones where application is running is gone, the application (a1) moves to one of the other zones and can access the data from the available replicas.
Kubernetes Monitoring and Troubleshooting using Weavescope- Kubernetes Meetup...OpenEBS
Why monitoring is important?
Application exceptions
Server CPU usage
Server memory usage
Storage Spikes
What is Kubernetes?
Container orchestration tool
Comprises of :
Pods
Services
Controllers
Persistent Volumes
OpenEBS Visualization and Monitoring using Weave-scope - Contributors Meet 1s...OpenEBS
This topic was presented by Satyam - an OpenEBS Contributor at Weekly Contributors Meet held on 1st June.
Weave Scope helps microservices architecture by
Networking of microservices
Visualization of infra
Ease of monitoring and logging
Better view of system configuration
Storage view
BDD Testing Using Godog - Bangalore Golang Meetup # 32OpenEBS
BDD uses natural language to describe the "desired behaviour" of the system, that can be understood by both the developer and the customer
Demo of an existing BDD application using Godog predominantly used with golang
Container Attached Storage - Chennai Kubernetes Meetup #2 - April 21st 2018OpenEBS
Container Attached Storage or CAS is a truly cloud native software architecture for applications running in containers. In CAS, the storage software itself is containerized and hence gets the advantages of being a micro service. In CAS architecture, each storage volume gets it's own storage controller running completely in user space and attains the maximum agility and policy granularity.
The slides were presented @ Kubernetes Meetup Bangalore held on 26th May - 2018 by Sathyam Zode - An OpenEBS Contributor and Software Engineer at MayaData .
There is a huge need to containerize the Stateful applications in today's world of Docker and Kubernetes. These stateful applications need a storage architecture that is truly cloud native. Container Attached Storage or CAS is a truly cloud native software architecture for applications running in containers. In CAS, the storage software itself is containerized and hence gets the advantages of being a micro service. In CAS architecture, each storage volume gets it's own storage controller running completely in user space and attains the maximum agility and policy granularity.
Containerized storage for CI/CD Pipeline and DevOps - why, how, huh? What is wrong w/ stateful workloads on containers today? What is happening at the Linux kernel to improve the security of containers as a platform FOR storage? Could containers and Kubernetes become the foundations of a new approach to storage? How to deal with storage in multi-cloud environment. Quick view into the OpenEBS project.
Murat Karslioglu is a serial entrepreneur, technologist, and startup geek with over 15 years of experience in storage, distributed systems, and enterprise software development. He is currently the VP of Solutions and Technology at CloudByte / OpenEBS, a startup developing open-source containerized storage for containers.
Prior to joining CloudByte, Murat has worked at Hewlett Packard Enterprise / 3PAR Storage in various advanced development projects including storage file stack performance optimization and the storage management stack for HPE’s Hyper-converged solution. Murat holds a Bachelor’s Degree in Industrial Engineering from the Sakarya University.
Containerized Storage for Containers- Kubernetes LA Meetup , July 2017OpenEBS
The Containerized Storage for Containers(OpenEBS) slides were presented in the Kubernetes Meetup , LA on July 18th , 2017. OpenEBS delivers storage and storage services to containerized environments. OpenEBS allows stateful workloads to be managed more like stateless containers. OpenEBS storage services include: per container (or pod) QoS SLAs, tiering and replica policies across AZs and environments, and predictable and scalable performance. To know more about OpenEBS visit our website www.openebs.io
Dynamic Instrumentation- OpenEBS Golang Meetup July 2017OpenEBS
The slides were presented by Jeffry Molanus who is the CTO of OpenEBS in Golang Meetup. OpenEBS is an open source cloud native storage. OpenEBS delivers storage and storage services to containerized environments. OpenEBS allows stateful workloads to be managed more like stateless containers. OpenEBS storage services include: per container (or pod) QoS SLAs, tiering and replica policies across AZs and environments, and predictable and scalable performance.Our vision is simple: let’s let storage and storage services for persistent workloads be so fully integrated into the environment and hence managed automatically that is almost disappears into the background as just yet another infrastructure service that works.
The OpenEBS Hangout #2 was held on 29th of June where we discussed how to deploy Jupyter Notebooks on OpenEBS. We also showed hyperconverged version of OpenEBS
A Comprehensive Look at Generative AI in Retail App Testing.pdfkalichargn70th171
Traditional software testing methods are being challenged in retail, where customer expectations and technological advancements continually shape the landscape. Enter generative AI—a transformative subset of artificial intelligence technologies poised to revolutionize software testing.
Designing for Privacy in Amazon Web ServicesKrzysztofKkol1
Data privacy is one of the most critical issues that businesses face. This presentation shares insights on the principles and best practices for ensuring the resilience and security of your workload.
Drawing on a real-life project from the HR industry, the various challenges will be demonstrated: data protection, self-healing, business continuity, security, and transparency of data processing. This systematized approach allowed to create a secure AWS cloud infrastructure that not only met strict compliance rules but also exceeded the client's expectations.
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?XfilesPro
Worried about document security while sharing them in Salesforce? Fret no more! Here are the top-notch security standards XfilesPro upholds to ensure strong security for your Salesforce documents while sharing with internal or external people.
To learn more, read the blog: https://www.xfilespro.com/how-does-xfilespro-make-document-sharing-secure-and-seamless-in-salesforce/
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Strategies for Successful Data Migration Tools.pptxvarshanayak241
Data migration is a complex but essential task for organizations aiming to modernize their IT infrastructure and leverage new technologies. By understanding common challenges and implementing these strategies, businesses can achieve a successful migration with minimal disruption. Data Migration Tool like Ask On Data play a pivotal role in this journey, offering features that streamline the process, ensure data integrity, and maintain security. With the right approach and tools, organizations can turn the challenge of data migration into an opportunity for growth and innovation.
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Hivelance Technology
Cryptocurrency trading bots are computer programs designed to automate buying, selling, and managing cryptocurrency transactions. These bots utilize advanced algorithms and machine learning techniques to analyze market data, identify trading opportunities, and execute trades on behalf of their users. By automating the decision-making process, crypto trading bots can react to market changes faster than human traders
Hivelance, a leading provider of cryptocurrency trading bot development services, stands out as the premier choice for crypto traders and developers. Hivelance boasts a team of seasoned cryptocurrency experts and software engineers who deeply understand the crypto market and the latest trends in automated trading, Hivelance leverages the latest technologies and tools in the industry, including advanced AI and machine learning algorithms, to create highly efficient and adaptable crypto trading bots
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
4. Maya: Cross-cloud control plane
○ Visibility, automation, collaboration, and, over
time, learning via machine learning
○ OpenEBS users can subscribe to a free
version and then are upsold to a subscription
that includes OpenEBS enterprise support
OpenEBS: Containerized Storage for
Containers
○ Open source software that allows each
workload - and DevOps team - to have their
own storage controller
10. Converged: Kubernetes + OpenEBS
K8s Master
Minion
POD
Container
Container
ContainerKubelet
POD
Container
Container
ContainerKubelet
POD
Container
Container
ContainerKubelet
Minion
POD
Container
Container
ContainerKubelet
POD
Container
Container
ContainerKubelet
POD
Container
Container
ContainerKubelet
Minion
POD
Container
Container
ContainerKubelet
POD
Container
Container
ContainerKubelet
POD
Container
Container
ContainerKubelet
etcd
APIs
Cntrl
Schld
OpenEBSAPIs Schld
Data Containers run in PODs on physical machines - an entire enterprise class storage controller
Data Containers mean every workload - and every per app team - has their own controller
OpenEBS runs on the Master; delivers services such as: APIs, the storage scheduler, analytics & others
11. How to get started ?
On your kubemaster:
kubemaster~: kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml
kubemaster~: kubectl apply -f percona.yaml
In the application yaml, choose the OpenEBS storage class, setup the policies, and launch the application
kubemaster~: kubectl get pods | grep pvc
pvc-8a9fc4b1-d838-11e7-9caa-42010a8000a7-ctrl-696530238-ngrgj 2/2 Running 0 36s
pvc-8a9fc4b1-d838-11e7-9caa-42010a8000a7-rep-3408218758-2ldzv 1/1 Running 0 36s
pvc-8a9fc4b1-d838-11e7-9caa-42010a8000a7-rep-3408218758-6mwj5 1/1 Running 0 36s
OpenEBS provisions storage controller with requested number of replicas, volume is bound and ready
16. What is coming in OpenEBS 0.6 ?● OpenEBS provisioner will support more storage spec from k8s
○ PV Resource Policy (Quota, #PVCs, etc., )
○ Volume Resize
○ Volume Snapshots
○ Block Volume Claims
● Disk Monitoring and Alerts
● Refactor Storage Policy specification as CRDs
● Support OpenEBS Upgrades via kubectl
● Enhance the debuggability
● Enhance CI with platform testing with OpenShift/CentOS, CoreOS, Rancher
18. cStore can store up to 2^128 bits can achieve millions of IOPS with usec latency
what cStor is not
● A distributed file system typically needed for capacity and
performance scaling; you can't have one without the other
○ hard to manage in production (do not want a storage team
● Volumes are typically small, GBs certainly not PBs
○ no need to scale capacity using complex distributed algorithms
● What about performance?
○ NMVe devices widely available in the cloud
○ single NMVe can do up to 400K iops; 3d Xpoint on its way
● Cloud native application have built in scalability
○ no need to scale a monolith storage system by adding more
drives to “raid groups”
19. Reimagined how storage should for cloud native apps on prem and in the cloud
What is cStor
● New storage engine that brings enterprise class features in containers for
containers
○ snapshot, clones, compression, replication, data integrity, …..
● Key enabler for cMotion (demo)
○ the ability to move data efficiently and incrementally c2c
● Always consistent on disk (transactions)
● Data integrity and encryption, crucial for the clouds deployments
● Online expansion of existing volumes (resize)
● Cloud native design vs cloud washed
○ build from the ground up vs an existing solution with container lipstick
20. Copy on Write
Under the hood of cStor
controller
cStor
node1
cStor
node2
cStor
node3
iSCSI, iSER, NMVEoF, NFS(?)
● Controller serves out the blocks to the
application
○ defined in YAML, deployed by maya and
k8s
● Based on replication level, the controller
forwards the IO to the replicas (cStor)
● cStor is a transactional; which is always
consistent on disk.
● Copy on Write (CoW) data never gets
overwritten but written to unused block
Also a container!
21. Atomic updates, data always consistent on disk -- cStor itself is stateless
Transactions
● Each write is assigned a transaction
● Transactions are batched in to transactions groups (optimal bandwidth)
● Latest transaction number points to the “live” data
● Transaction numbers are updated atomically which means that all
writes in the group have succeeded or failed
● A snapshot is a reference to an old transaction (and its data)
○ quick scan of newly written blocks since last transaction
● cMotion; send the blocks that have changed between two transactions
● All form nice and comfortable user space
○ no kernel dependencies (needed for c2c)
○ no kernel taints
22. Hardware trends enforce a change in the way we do things
Storage performance challenges
● How to achieve high performance numbers from within user space?
○ copyin/copyout of data between kernel and user is expensive
○ context switches
● With current HW trends, the kernel becomes the bottleneck
○ white label boxes 1U, serving out 17mil IOPS (!!!)
● Low latency SSDs and 100GB network cards become the norm
● 10GB nic
○ 14.888e6 64 bytes per second; CPU has only a couple of cycles
per packet per NIC
● Frequency remains relatively the same, core count goes up
○ we’ve got cores to spare
23. HW performance kernel vs user
Its raining IOPS
Source : https://software.intel.com/en-us/articles/accelerating-your-nvme-drives-with-spdk
24. Achieve higher performance in user space
● Solution; bypass the kernel subsystem all together, running
everything in user space
● Instead of doing network or disk IO to the kernel, we submit the IO to
another container which is direct access to the HW resources (IOC)
○ map NIC rings to user space
○ map PCI bars from NMVe devices
○ lockless and message passing between cores
● Poll Mode Drivers (PMD)
○ 100% CPU
● Borrow from VM technology to construct interfaces between
containers
○ VHOST and VIRTIO-SCSI
25. Summary
● cStor provides enterprise class like features like your friendly
neighbourhood <insert vendor> storage system
● Provides data integrity features missing natively in the Linux Kernel
● Provides the ability to efficiently work with data by use of COW
● Bypasses kernel for IO to achieve higher performance than kernel
● Cloud native design; using cloud native paradigms to develop and
deploy
● Removes friction between developers and storage admins
QUESTIONS?
27. MayaOnline cMotion Demo setup overview
Jenkins
Pod
GCP Zone: US-Central
K8s Cluster: austin-cicd
User’s
CI/CD
GCP Zone: Europe East
K8s Cluster: Denmark-
cicd
AWS Zone: US East
K8s Cluster: mule-master
1
Part 1: Show the CICD
setup working with
Jenkins and github
28. MayaOnline cMotion Demo setup overview
Jenkins
Pod
GCP Zone: US-Central
K8s Cluster: austin-cicd
User’s
CI/CD
GCP Zone: Europe East
K8s Cluster: Denmark-
cicd
AWS Zone: US East
K8s Cluster: mule-master
1
Part 1: Show the CICD
setup working with
Jenkins and github
2
Part 2: Move Jenkins pod
to Denmark k8s cluster
and show the github
CICD working
Jenkins
Pod
29. MayaOnline cMotion Demo setup overview
Jenkins
Pod
GCP Zone: US-Central
K8s Cluster: austin-cicd
User’s
CI/CD
GCP Zone: Europe East
K8s Cluster: Denmark-
cicd
AWS Zone: US East
K8s Cluster: mule-master
1
Part 1: Show the CICD
setup working with
Jenkins and github
2
Part 2: Move Jenkins pod
to GCP Denmark k8s
cluster and show the
github CICD working
Jenkins
Pod
3
Part 3: Move Jenkins pod
to AWS k8s cluster and
show the github CICD
working
30. MayaOnline cMotion Demo setup overview
Jenkins
Pod
GCP Zone: US-Central
K8s Cluster: austin-cicd
User’s
CI/CD
GCP Zone: Europe East
K8s Cluster: Denmark-
cicd
AWS Zone: US East
K8s Cluster: mule-master
1
Part 1: Show the CICD
setup working with
Jenkins and github
2
Part 2: Move Jenkins pod
to GCP Denmark k8s
cluster and show the
github CICD working
Jenkins
Pod
3
Part 3: Move Jenkins pod
to AWS k8s cluster and
show the github CICD
working
4
Part 4: Move Jenkins pod
to GCP austin cluster and
show the github CICD
working
32. Container Attached Storage = DAS++
DAS
Benefits:
Simple
Ties application to storage
Predictable for capacity
planning
App deals with resiliency
Can be faster
Concerns:
Under-utilized hardware
○ 10% or less utilization
Wastes data center
Difficult to manage
Lacks storage features
Cannot be repurposed - made
for one workload
Does not support mobility of
workloads via containers
Cross cloud impossible
OpenEBS = “CAS”
✓ Simple
✓ No new skills required
✓ Per microservice storage policy
✓ Data protection & snapshots
✓ Reduces cloud vendor lock-in
✓ Eliminates storage vendor lock-in
✓ Highest possible efficiency
✓ Large & growing OSS community
✓ Natively cross cloud
✓ Uses proven code - ZFS & Linux
✓ Maya -> ML based analytics &
tuning
“YASS”: Distributed
Benefits:
Centralized management
Greater density and efficiency
Storage features such as:
○ Data protection
○ Snapshots for versioning
Concerns:
Additional complexity
Enormous blast radius
Expensive
Requires storage engineering
Challenged by container dynamism
No per microservice storage policy
I/O blender impairs performance
Locks customers into vendor
Cross cloud impossible
CAS
DAS
Distributed
Editor's Notes
Ask questions - what good is a plan to you? What are you hoping to get out of this session?
Hyperconverged
Hyperconverged with CO
Smaller Blast radius with micro-services-like Storage Controller Architecture
Seamless management interface - similar to Kurbernetes (kubernetes itself)
Extends the capabilities of CO with Storage Management
Benefits of Locally Attached Storage and High Availability provided via the Synchronous Replication
Easy to Migrate across Nodes, Cluster, Infra ( No Cloud Vendor Lock-in)
So first, this may seem odd, but I want to explain what cstor is not, but more importantly why not.
Storage veterans, so features that we believe are needed
not everything on the list is done yet, but certainly in the pipes