Solving k8s persistent workloads using k8s DevOps styleMayaData
Solving k8s persistent workloads
using k8s DevOps style. Presented at Container_stack-Zurich-2019
-How Hardware trends enforce a change in the way we do things
-Storage limitations bubble up
-Infrastructure as code
Latest (storage IO) patterns for cloud-native applications OpenEBS
Applying micro service patterns to storage giving each workload its own Container Attached Storage (CAS) system. This puts the DevOps persona within full control of the storage requirements and brings data agility to k8s persistent workloads. We will go over the concept and the implementation of CAS, as well as its orchestration.
Communication between Microservices is inherently unreliable. These integration points may produce cascading failures, slow responses, service outages. We will walk through stability patterns like timeouts, circuit breaker, bulkheads and discuss how they improve stability of Microservices.
JCConf 2016 - Cloud Computing Applications - Hazelcast, Spark and IgniteJoseph Kuo
This session aims to establish applications running against distributed and scalable system, or as we know cloud computing system. We will introduce you not only briefing of Hazelcast but also deeper kernel of it, and how it works with Spark, the most famous Map-reduce library. Furthermore, we will introduce another in-memory cache called Apache Ignite and compare it with Hazelcast to see what's the difference between them. In the end, we will give a demonstration showing how Hazelcast and Spark work together well to form a cloud-base service which is distributed, flexible, reliable, available, scalable and stable. You can find demo code here: https://github.com/CyberJos/jcconf2016-hazelcast-spark
https://cyberjos.blog/java/seminar/jcconf-2016-cloud-computing-applications-hazelcast-spark-and-ignite/
From distributed caches to in-memory data gridsMax Alexejev
A brief introduction into modern caching technologies, starting from distributed memcached to modern data grids like Oracle Coherence.
Slides were presented during distributed caching tech talk in Moscow, May 17 2012.
Solving k8s persistent workloads using k8s DevOps styleMayaData
Solving k8s persistent workloads
using k8s DevOps style. Presented at Container_stack-Zurich-2019
-How Hardware trends enforce a change in the way we do things
-Storage limitations bubble up
-Infrastructure as code
Latest (storage IO) patterns for cloud-native applications OpenEBS
Applying micro service patterns to storage giving each workload its own Container Attached Storage (CAS) system. This puts the DevOps persona within full control of the storage requirements and brings data agility to k8s persistent workloads. We will go over the concept and the implementation of CAS, as well as its orchestration.
Communication between Microservices is inherently unreliable. These integration points may produce cascading failures, slow responses, service outages. We will walk through stability patterns like timeouts, circuit breaker, bulkheads and discuss how they improve stability of Microservices.
JCConf 2016 - Cloud Computing Applications - Hazelcast, Spark and IgniteJoseph Kuo
This session aims to establish applications running against distributed and scalable system, or as we know cloud computing system. We will introduce you not only briefing of Hazelcast but also deeper kernel of it, and how it works with Spark, the most famous Map-reduce library. Furthermore, we will introduce another in-memory cache called Apache Ignite and compare it with Hazelcast to see what's the difference between them. In the end, we will give a demonstration showing how Hazelcast and Spark work together well to form a cloud-base service which is distributed, flexible, reliable, available, scalable and stable. You can find demo code here: https://github.com/CyberJos/jcconf2016-hazelcast-spark
https://cyberjos.blog/java/seminar/jcconf-2016-cloud-computing-applications-hazelcast-spark-and-ignite/
From distributed caches to in-memory data gridsMax Alexejev
A brief introduction into modern caching technologies, starting from distributed memcached to modern data grids like Oracle Coherence.
Slides were presented during distributed caching tech talk in Moscow, May 17 2012.
Do you need to scale your application, share data across cluster, perform massive parallel processing on many JVMs or maybe consider alternative to your favorite NoSQL technology? Hazelcast to the rescue! With Hazelcast distributed development is much easier. This presentation will be useful to those who would like to get acquainted with Hazelcast top features and see some of them in action, e.g. how to cluster application, cache data in it, partition in-memory data, distribute workload onto many servers, take advantage of parallel processing, etc.
Presented on JavaDay Kyiv 2014 conference.
From cache to in-memory data grid. Introduction to Hazelcast.Taras Matyashovsky
This presentation:
* covers basics of caching and popular cache types
* explains evolution from simple cache to distributed, and from distributed to IMDG
* not describes usage of NoSQL solutions for caching
* is not intended for products comparison or for promotion of Hazelcast as the best solution
NoSQL databases, the CAP theorem, and the theory of relativityLars Marius Garshol
A presentation showing how the CAP theorem causes NoSQL databases to have BASE semantics. That is, they don't support ACID consistency. Then shows how CAP is related to Einstein's theory of relativity. And finally shows how Google Spanner and F1 provide ACID that scales.
How LinkedIn uses memcached, a spoonful of SOA, and a sprinkle of SQL to scaleLinkedIn
This is one of two presentations given by LinkedIn engineers at Java One 2009.
This presentation was given by David Raccah and Dhananjay Ragade from LinkedIn.
For more information, check out http://blog.linkedin.com/
Introduction to failover clustering with sql serverEduardo Castro
In this presentation we review the basic requirements to install a SQL Server Failover Cluster.
Regards,
Eduardo Castro Martinez
http://ecastrom.blogspot.com
http://comunidadwindows.org
WANdisco Non-Stop Hadoop: PHXDataConference Presentation Oct 2014 Chris Almond
Hadoop has quickly evolved into the system of choice for storing and processing Big Data, and is now widely used to support mission-critical applications that operate within a ‘data lake’ style infrastructures. A critical requirement of such applications is the need for continuous operation even in the event of various system failures. This requirement has driven adoption of multi-data center Hadoop architectures, a.k.a geo-distributed or global Hadoop. In this session we will provide a brief introduction to WANdisco, then dig into how our Non-Stop Hadoop solution addresses real world use cases, and also a show live demonstration of Non-Stop namenode operation across two WAN connected hadoop clusters.
10 Tricks to Ensure Your Oracle Coherence Cluster is Not a "Black Box" in Pro...SL Corporation
Configuration of Oracle Coherence can be tricky. While Coherence provides highly valuable in-memory caching and parallel processing features, things don’t always go as planned, and changes can be extremely difficult to make once you’re in production. SL’s Founder and CTO, Tom Lubinski covers 10 things you can do to ensure your Coherence cluster is easy to support in production.
OpenEBS; asymmetrical block layer in user-space breaking the million IOPS bar...MayaData
Presented at FOSDEM 2019
K8s as a universal control plane to deploy containerised applications • Public cloud is moving on premises (GKE, Outpost) • K8s capable of doing more then containers due to controllers (VMs)
SpringPeople - Introduction to Cloud ComputingSpringPeople
Cloud computing is no longer a fad that is going around. It is for real and is perhaps the most talked about subject. Various players in the cloud eco-system have provided a definition that is closely aligned to their sweet spot –let it be infrastructure, platforms or applications.
This presentation will provide an exposure of a variety of cloud computing techniques, architecture, technology options to the participants and in general will familiarize cloud fundamentals in a holistic manner spanning all dimensions such as cost, operations, technology etc
Do you need to scale your application, share data across cluster, perform massive parallel processing on many JVMs or maybe consider alternative to your favorite NoSQL technology? Hazelcast to the rescue! With Hazelcast distributed development is much easier. This presentation will be useful to those who would like to get acquainted with Hazelcast top features and see some of them in action, e.g. how to cluster application, cache data in it, partition in-memory data, distribute workload onto many servers, take advantage of parallel processing, etc.
Presented on JavaDay Kyiv 2014 conference.
From cache to in-memory data grid. Introduction to Hazelcast.Taras Matyashovsky
This presentation:
* covers basics of caching and popular cache types
* explains evolution from simple cache to distributed, and from distributed to IMDG
* not describes usage of NoSQL solutions for caching
* is not intended for products comparison or for promotion of Hazelcast as the best solution
NoSQL databases, the CAP theorem, and the theory of relativityLars Marius Garshol
A presentation showing how the CAP theorem causes NoSQL databases to have BASE semantics. That is, they don't support ACID consistency. Then shows how CAP is related to Einstein's theory of relativity. And finally shows how Google Spanner and F1 provide ACID that scales.
How LinkedIn uses memcached, a spoonful of SOA, and a sprinkle of SQL to scaleLinkedIn
This is one of two presentations given by LinkedIn engineers at Java One 2009.
This presentation was given by David Raccah and Dhananjay Ragade from LinkedIn.
For more information, check out http://blog.linkedin.com/
Introduction to failover clustering with sql serverEduardo Castro
In this presentation we review the basic requirements to install a SQL Server Failover Cluster.
Regards,
Eduardo Castro Martinez
http://ecastrom.blogspot.com
http://comunidadwindows.org
WANdisco Non-Stop Hadoop: PHXDataConference Presentation Oct 2014 Chris Almond
Hadoop has quickly evolved into the system of choice for storing and processing Big Data, and is now widely used to support mission-critical applications that operate within a ‘data lake’ style infrastructures. A critical requirement of such applications is the need for continuous operation even in the event of various system failures. This requirement has driven adoption of multi-data center Hadoop architectures, a.k.a geo-distributed or global Hadoop. In this session we will provide a brief introduction to WANdisco, then dig into how our Non-Stop Hadoop solution addresses real world use cases, and also a show live demonstration of Non-Stop namenode operation across two WAN connected hadoop clusters.
10 Tricks to Ensure Your Oracle Coherence Cluster is Not a "Black Box" in Pro...SL Corporation
Configuration of Oracle Coherence can be tricky. While Coherence provides highly valuable in-memory caching and parallel processing features, things don’t always go as planned, and changes can be extremely difficult to make once you’re in production. SL’s Founder and CTO, Tom Lubinski covers 10 things you can do to ensure your Coherence cluster is easy to support in production.
OpenEBS; asymmetrical block layer in user-space breaking the million IOPS bar...MayaData
Presented at FOSDEM 2019
K8s as a universal control plane to deploy containerised applications • Public cloud is moving on premises (GKE, Outpost) • K8s capable of doing more then containers due to controllers (VMs)
SpringPeople - Introduction to Cloud ComputingSpringPeople
Cloud computing is no longer a fad that is going around. It is for real and is perhaps the most talked about subject. Various players in the cloud eco-system have provided a definition that is closely aligned to their sweet spot –let it be infrastructure, platforms or applications.
This presentation will provide an exposure of a variety of cloud computing techniques, architecture, technology options to the participants and in general will familiarize cloud fundamentals in a holistic manner spanning all dimensions such as cost, operations, technology etc
Data Lake and the rise of the microservicesBigstep
By simply looking at structured and unstructured data, Data Lakes enable companies to understand correlations between existing and new external data - such as social media - in ways traditional Business Intelligence tools cannot.
For this you need to find out the most efficient way to store and access structured or unstructured petabyte-sized data across your entire infrastructure.
In this meetup we’ll give answers on the next questions:
1. Why would someone use a Data Lake?
2. Is it hard to build a Data Lake?
3. What are the main features that a Data Lake should bring in?
4. What’s the role of the microservices in the big data world?
Webinar: Overcoming the Storage Challenges Cassandra and Couchbase CreateStorage Switzerland
NoSQL databases like Cassandra and Couchbase are quickly becoming key components of the modern IT infrastructure. But this modernization creates new challenges – especially for storage. Storage in the broad sense. In-memory databases perform well when there is enough memory available. However, when data sets get too large and they need to access storage, application performance degrades dramatically. Moreover, even if enough memory is available, persistent client requests can bring the servers to their knees.
Join Storage Switzerland and Plexistor where you will learn:
1. What is Cassandra and Couchbase?
2. Why organizations are adopting them?
3. What are the storage challenges they create?
4. How organizations attempt to workaround these challenges.
5. How to design a solution to these challenges instead of a workaround.
Today, most any application can be “Dockerized.” However, there are special challenges when deploying a distributed application such as Spark on containers. This session will describe how to overcome these challenges in deploying Spark on Docker containers, with many practical tips and techniques for running Spark in a container environment.
Containers are typically used to run stateless applications on a single host. There are significant real-world enterprise requirements that need to be addressed when running a stateful, distributed application in a secure multi-host container environment.
There are decisions that need to be made concerning which tools and infrastructure to use. There are many choices with respect to container managers, orchestration frameworks, and resource schedulers that are readily available today and some that may be available tomorrow including:]
• Mesos
• Kubernetes
• Docker Swarm
Each has its own strengths and weaknesses; each has unique characteristics that may make it suitable, or unsuitable, for Spark. Understanding these differences is critical to the successful deployment of Spark on Docker containers.
This session will describe the work done by the BlueData engineering team to run Spark inside containers, on a distributed platform, including the evaluation of various orchestration frameworks and lessons learned. You will learn how to apply practical networking and storage techniques to achieve high performance and agility in a distributed, container environment.
Speaker
Thomas Phelan, Chief Architect, Blue Data, Inc
Lessons Learned Running Hadoop and Spark in Docker ContainersBlueData, Inc.
Many initiatives for running applications inside containers have been scoped to run on a single host. Using Docker containers for large-scale production environments poses interesting challenges, especially when deploying distributed big data applications like Apache Hadoop and Apache Spark. This session at Strata + Hadoop World in New York City (September 2016) explores various solutions and tips to address the challenges encountered while deploying multi-node Hadoop and Spark production workloads using Docker containers.
Some of these challenges include container life-cycle management, smart scheduling for optimal resource utilization, network configuration and security, and performance. BlueData is "all in” on Docker containers—with a specific focus on big data applications. BlueData has learned firsthand how to address these challenges for Fortune 500 enterprises and government organizations that want to deploy big data workloads using Docker.
This session by Thomas Phelan, co-founder and chief architect at BlueData, discusses how to securely network Docker containers across multiple hosts and discusses ways to achieve high availability across distributed big data applications and hosts in your data center. Since we’re talking about very large volumes of data, performance is a key factor, so Thomas shares some of the storage options implemented at BlueData to achieve near bare-metal I/O performance for Hadoop and Spark using Docker as well as lessons learned and some tips and tricks on how to Dockerize your big data applications in a reliable, scalable, and high-performance environment.
http://conferences.oreilly.com/strata/hadoop-big-data-ny/public/schedule/detail/52042
Big-Data-as-a-Service (BDaaS) in an enterprise environment requires meeting the often contradictory goals of (1) providing your data scientists, analysts, and data engineers with a self-service consumption model; (2) delivering agile and scalable on-demand infrastructure for the rapidly evolving ecosystem of big data frameworks and application software; while (3) ensuring enterprise-grade capabilities for isolation, security, monitoring, etc.
In this presentation at our BDaaS meetup in Santa Clara, Tom Phelan (chief architect and co-founder of BlueData) reviewed these goals and how to resolve the potential contradictions. He also discussed the infrastructure, application, user experience, security, and maintainability considerations required before selecting (or designing and building) a Big-Data-as-a-Service platform for an enterprise big data deployment.
More info on this BDaaS meetup can be found at: http://www.meetup.com/Big-Data-as-a-Service/events/233999817
Jay Kreps on Project Voldemort Scaling Simple Storage At LinkedInLinkedIn
Jay Kreps on Project Voldemort Scaling Simple Storage At LinkedIn. This was a presentation made at QCon 2009 and is embedded on LinkedIn's blog - http://blog.linkedin.com/
Similar to Container Attached Storage with OpenEBS - CNCF Paris Meetup (20)
MayaData Datastax webinar - Operating Cassandra on Kubernetes with the help ...MayaData Inc
In this webinar experts from DataStax - the lead developer of Cassandra - and from MayaData - the lead developer of OpenEBS and LitmusChaos - will discuss and demonstrate ways to ensure the ease of use and resilience of Cassandra on Kubernetes.
Topics to be discussed and demonstrated include:
Provisioning underlying storage - how to make it consistent irrespective of the underlying hardware or cloud? Are there are ever reasons to have the storage replicate across nodes or is dynamic LocalPV the best choice in all cases?
Cass Operator - DataStax Kubernetes Operator for Apache Cassandra
Resilience - how to proactively assess the overall environment including the underlying Kubernetes with the help of Litmus
Webinar: Data Protection for KubernetesMayaData Inc
In this webinar, we will back-up many live workloads to the Cloudian Hyperstore from a Kubernetes environment running on a particular cloud. We will demonstrate the value of Cloudian’s WORM capabilities to show how workloads and their data can be protected from ransomware attacks. Later, we will recover workloads from the Cloudian HyperStore to another cloud vendor. We will also demonstrate streaming back-ups for use in cloud and hardware switch overs and other use cases.
Kubera from MayaData is the first solution to extend the per workload management of data offered by Container Attached Storage to back-ups and disaster recovery. Kubera is often used by small teams to establish and manage back-up policies whereby data is backed up to S3 compatible object storage. Kubera can also be used to provide a comprehensive view across all workloads of back-up and retention policies and to enable back-ground cloud migration and disaster recovery.
Kubera is a SaaS platform - also available on-premise - that simplifies the use of Kubernetes as a data plane and that is free for individual usage.
Core capabilities include:
Visualization of a Kubernetes environment, including stateful workloads and the resources serving them
Data resilience capabilities, such as cross availability zone configuration, crash-consistent consistent back-ups, pre-staged disaster recovery, chaos test integration, and more
Off cluster logging and alerting
Autoconfiguration and management of OpenEBS Enterprise Edition
Integrated support services from MayaData
Webinar: OpenEBS - Still Free and now FASTEST Kubernetes storageMayaData Inc
Webinar Session - https://youtu.be/_5MfGMf8PG4
In this webinar, we share how the Container Attached Storage pattern makes performance tuning more tractable, by giving each workload its own storage system, thereby decreasing the variables needed to understand and tune performance.
We then introduce MayaStor, a breakthrough in the use of containers and Kubernetes as a data plane. MayaStor is the first containerized data engine available that delivers near the theoretical maximum performance of underlying systems. MayaStor performance scales with the underlying hardware and has been shown, for example, to deliver in excess of 10 million IOPS in a particular environment.
Save 60% of Kubernetes storage costs on AWS & others with OpenEBSMayaData Inc
With features like thin provisioning, per workload replication and snapshots, using OpenEBS can lower your storage TCO on any Kubernetes cloud by up to 60%. In this webinar you will see with in depth examples of the method a MayaData OpenEBS Enterprise customer used to save $ 75,000 a month.
Webinar: Using Litmus Chaos Engineering and AI for auto incident detectionMayaData Inc
Chaos engineering tools offer a great way to test an application’s resiliency in a Kubernetes deployment. But chaos experiments can induce failure modes that have never been seen before, causing incidents to slip through existing alert rules.
Detecting these failures can be tricky, and Zebrium solves this with unassisted machine learning.
Our interactive panel discusses, in the context of a demo, a set of Litmus chaos engineering experiments against a distributed Kubernetes app and will use Zebrium Autonomous Log Monitoring to auto-detect incidents and provide an indication of the root cause.
Webinar: Building a multi-cloud Kubernetes storage on GitLabMayaData Inc
In this webinar, we talk about how to set up redundant and highly available storage for your repos to make your key repositories easier to deploy, more reliable, and easier to back up or move to a different cloud. We reviewed the current practices for highly-available CI/CD and showcased how there’s a better way to do it with OpenEBS.
OpenEBS Technical Workshop - KubeCon San Diego 2019MayaData Inc
Know how to navigate the journey to cloud-native data management with lessons learned and best practices to help you deploy Kubernetes, storage, and data management with confidence.
OPENEBS 0.8.1 RELEASE
We are glad to announce that OpenEBS 0.8.1 is released. It has significant stability improvements and few important features. Most of the issues fixed in this release are in response to the feedback received from the community.
In this webinar, we take you through some important highlights
What's new in 0.8.1 and roadmap
• How to make the most of your cStor
• MayaOnline: New Features and Capabilities
How to Run Containerized Enterprise SQL Applications in the Cloud with NuoDB ...MayaData Inc
Deploying an enterprise SQL database across geographically located OpenShift or Kubernetes clusters can be challenging. These deployments often require zero-downtime, ANSI standard SQL, ACID-compliant transactions, seamless day-2 operations, and highly performant and durable persistent storage systems. How can your organization easily deploy container-native storage with a distributed SQL database to deliver containerized apps in the cloud?
In this webinar, NuoDB and OpenEBS (MayaData) guide you as you build containerized apps that check these critical boxes:
[✓] Always on
[✓] At scale
[✓] High-performance persistent storage
D2iQ (formerly Mesosphere) Konvoy provides a standalone production-ready Kubernetes cluster with best-in-class components for operation and lifecycle management and Kommander to supercharge Day 2 operations.
Based on the leading open source Container Attached Storage software, MayaData OpenEBS Enterprise Platform reduces the risk and increases the agility of running stateful applications on Kubernetes.
In this webinar, experts from MayaData and D2iQ will talk about the benefits of using Konvoy and OpenEBS as part of an integrated solution on any cloud infrastructure. What makes Konvoy unique, it’s deployment and the typical use cases and show you how OpenEBS can be used to deploy and update Konvoy and also how OpenEBS can be deployed on Konvoy to simplify the deployment, management and monitoring of stateful applications.
Before we get into Q&A, MayaData team will live demonstrate the combined solution of D2iQ Konvoy and OpenEBS.
Use GitLab with Chaos Engineering to Harden your Applications + OpenEBS 1.3 ...MayaData Inc
If you were not at the GitLab Commit conferences in New York and London, here’s an opportunity to attend our popular talk on using chaos engineering in Gitlab pipelines for faster hardening. As cloud native applications are coming to life faster than anyone could have imagined, the explosion of microservices empowers developers while also making it increasingly difficult to build pipelines that validate changes outside of their (or their SREs') control.
Chaos engineering has emerged as a way to introduce faults into systems to increase their resiliency and Litmus, part of OpenEBS Enterprise Platform, can shake out a lot of bugs.
We are also glad to announce that OpenEBS 1.3 has been released and we will review the new features added.
OpenEBS 1.1 has been released and is now ready for prime time. MayaData Agility Platform, the commercial distribution of OpenEBS combined with MayaOnline or MayaOnPrem and support, has also been updated and loaded with a lot of new features.
In this webinar, we will cover new features like:
Kubernetes Job for auto update of releases
Support for an alpha version of CSI driver with limited functionality for provisioning and de-provisioning of cStor volumes.
Additional platforms like Amazon Marketplace, Openshift Operator Hub, Rancher K3OS.
Bug fixes reported around the cStor volume, NDM, cStor Target, cStor Sparse tool and Jiva volumes.
Upgrade of the Valero plugin.
Link to view webinar:
https://go.mayadata.io/webinar/openebs-1.1release
Webinar:Kubecon Barcelona Update + OpenEBS 0.9 releaseMayaData Inc
OPENEBS NOW IN CNCF
In this webinar, MayaData team give a summary of what being a CNCF project means to the community and updates on the project engagement / contributor meeting schedules.
OPENEBS 0.9 RELEASE
New features introduced as part of the OpenEBS 0.9 release such as dynamic provisioning of Local PV, performance tuning, new functionalities for backup, HA and security and many other new features.
KUBECON ANNOUNCEMENTS
We will review some announcements made at KubeCon in Barcelona such as a new storage engine that brings NVMe to the world of Container Attached Storage and deploying OpenEBS at the edge and far-edge. We will also highlight some of the complementary ecosystem projects and of course, lessons learned during the week.
DEMO
Before we get into Q&A MayaData team will live demonstrate some of the features discussed in the release update.
Persistent Storage for stateful applications on Kubernetes made easy with Ope...MayaData Inc
In this webinar, Director of Community of Rancher Labs Jason van Brackel joins forces with Sr. Developer Advocate Patrick Hoolboom from MayaData to talk about benefits of OpenEBS and Rancher as a combined solution.
Rancher's multi-cluster Kubernetes management solution allows development teams to iterate fast, deploy efficiently and operate at scale. Kubernetes allows you to orchestrate containers that are highly available. However, in the case of container reschedule, Kubernetes does not provide a great set of primitives to manage your persistent data along with your application containers. In this webinar, we will present some of the challenges associated with managing persistent data in Kubernetes and how we can make day 2 operations easier to manage. We will briefly introduce the combined offering and talk about a couple of approaches to solving data persistence problems in multi-cloud environments with Rancher and OpenEBS. During the demos, we will showcase how we address data availability with OpenEBS.
We will also talk about project updates in the latest releases and preview of upcoming Kubecon announcements.
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamtakuyayamamoto1800
In this slide, we show the simulation example and the way to compile this solver.
In this solver, the Helmholtz equation can be solved by helmholtzFoam. Also, the Helmholtz equation with uniformly dispersed bubbles can be simulated by helmholtzBubbleFoam.
3. on premises Google packet.net
DMaaS
Analytics
Alerting
Compliance
Policies
Declarative Data Plane
A
P
I
Advisory
Chatbot
4. Resistance Is Futile
• K8s based on the original Google Borg paper
• Containers are the “unit” of management
• Mostly web based applications
• Typically the apps where stateless — if you agree there is such a thing
• In its most simplistic form k8s is a control loop
• Converge to the desired state based on declarative intent provided by the DevOps
persona
• Abstract away underlying compute cluster details and decouple apps from
infra structure: avoid lock-in
• Have developer focus on application deployment and not worry about the
environment it runs in
• HW independent (commodity)
6. Persistency in Volatile Environnements
• Containers storage is ephemeral; data is only stored during the life time of
the container(s)
• This either means that temporary data has no value or it can be regenerated
• Sharing data between containers is also a challenge — need to persist
• In the case of severless — the intermediate state between tasks is ephemeral
• The problem then: containers need persistent volumes in order to run state
full workloads
• While doing so: abstract away the underlying storage details and decouple
the data from the underlying infra: avoid lock-in
• The “bar” has been set in terms of expectation by the cloud providers i.e PD, EBS
• Volume available at multiple DCs and/or regions and replicated
7. Data Loss Is Almost Guaranteed
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
hostPath:
# directory location on host
path: /data
Unless…
8. Use a “Cloud” Disk
apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
# This GCE PD must already exist!
gcePersistentDisk:
pdName: my-data-disk
fsType: ext4
9. Evaluation and Progress
• In both cases we tie ourselves to a particular node — that defeats the agility
found natively in k8s and it failed to abstract away details
• We are cherrypicking pets from our herd
• anti pattern — easy to say and hard to avoid in some cases
• The second example allows us to mount (who?) the PV to different nodes
but requires volumes to be created prior to launching the workload
• Good — not great
• More abstraction through community efforts around Persistent Volumes
(PV) and Persistent Volume Claims (PVC) and CSI
• Container Storage Interface (CSI) to handle vendor specific needs before, in
example, mounting the volume
• Avoid wild fire of “volume plugins” or “drivers” in k8s main repo
11. Summary So Far
• Register a set of “mountable” things to the cluster (PVC)
• Take ownership of a “mountable” thing in the cluster (PV)
• Refer in the application to the PVC
• Dynamic provisioning; create ad-hoc PVCs when claiming something that
does not exist yet
• Remove the need to preallocate them (is that a good thing?)
• The attaching and detaching of volumes to nodes is standardised by means
of CSI which is an gRPC interface that handles the details of creating,
attaching, staging, destroying etc
• Vendor specific implementations are hidden from the users
12. The Basics — Follow the Workload
Node Node
POD
PVC
13. Problem Solved?
• How does a developer configure the PV such that it exactly has the features
that are required for that particular workload
• Number of replica’s, Compression, Snapshot and clones (opt in/out)
• How do we abstract away differences between storage vendors when
moving to/from private or public cloud?
• Differences in replication approaches — usually not interchangeable
• Abstract away access protocol and feature mismatch
• Provide cloud native storage type like “look and feel” on premises ?
• Don't throw away our million dollar existing storage infra
• GKE on premisses, AWS outpost — if you are not going to the cloud it will come to
you, resistance if futile
• Make data as agile as the applications that they serve
14. Data Gravity
• As data grows — it has the tendency to pull applications towards it (gravity)
• Everything will evolve around the sun and it dominates the planets
• Latency, throughput, IO blender
• If the sun goes super nova — all your apps circling it will be gone instantly
• Some solutions involve replicating the sun towards some other location in
the “space time continuum”
• It works — but it exacerbates the problem
17. Cloud Native Architecture?
• Applications have changed, and somebody forgot to tell storage
• Cloud native applications are —distributed systems themselves
• May use a variety of protocols to achieve consensus (Paxos, Gossip, etc)
• Is a distributed storage system still needed?
• Designed to fail and expected to fail
• Across racks, DC’s, regions and providers, physical or virtual
• Scalability batteries included
• HaProxy, Envoy, Nginx
• Datasets of individual containers relativity small in terms of IO and size
• Prefer having a collection of small stars over a big sun?
• The rise of cloud native languages such as Ballerina, Metaparticle etc
18. HW / Storage Trends
• Hardware trends enforce a change in the way we do things
• 40GbE and 100GbE are ramping up, RDMA capable
• NVMe and NVMe-OF (transport — works on any device)
• Increasing core counts — concurrency primitives built into languages
• Storage limitations bubble up in SW design (infra as code)
• “don’t do this because of that” — “don’t run X while I run my backup”
• Friction between teams creates “shadow it” — the (storage) problems start when
we move back from the dark side of the moon back into the sun
• “We simply use DAS —as there is nothing faster then that”
• small stars, that would work — no “enterprise features”?
• “they have to figure that out for themselves”
• Seems like storage is an agility anti-pattern?
20. The Persona Changed
• Deliver fast and frequently
• Infrastructure as code, declarative
intent, gitOps, chatOps
• K8s as the unified cross cloud
control plane (control loop)
• So what about storage? It has not
changed at all
21. The Idea
Manifests express intent
stateless
Container 1 Container 2 Container 3
stateful
Data Container Data Container Data Container
Any Server, Any Cloud Any Server, Any Cloud
container(n) container(n) container(n)
container(n) container(n) container(n)
22. Design Constraints
• Built on top of the substrate of Kubernetes
• That was a bet we made ~2 years ago that turned out to be right
• Not yet another distributed storage system; small is the new big
• Not to be confused with not scalable
• One on top of the other, an operational nightmare?
• Per workload: using declarative intent defined by the persona
• Runs in containers for containers — so it needs to run in user space
• Make volumes omnipresent — compute follows the storage?
• Where is the value? Compute or the data that feeds the compute?
• Not a clustered storage instance rather a cluster of storage instances
29. User Space and Performance
• NVMe as a transport is a game changer not just for its speed potential, but
also due to its relentless break away of the SCSI layer (1978)
• A Lot of similarities with Infini Band technology found in HPC for many years
(1999 as a result of a merger)
31. HW Changes Enforce A Change
• With these low latency devices CPUs are becoming the
bottleneck
• Post spectre/meltdown syscalls have become more expensive
then ever
35. K8S as a Control Loop
Kubelet
K8s
Master
YAML
+ -
Primary loop (k8s)
OP Sched
API
Servers
…..
36. -
+
Extending the K8S Control Loop
Kubeletk8s++
Adapt
YAML
+ -
RefMO
Primary loop (k8s)
Secondary loop (MOAC)
37. Raising the Bar — Automated Error Correction
CAS
FIO FIO FIO
replay blk IO pattern of various apps
kubectl scale up and down
DB
Regression
AI/ML
Logs Telemetry
Learn what failure
impacts app how
Declarative Data Plane
A
P
I