EasyHSM Overview

•

0 likes•529 views

This document discusses EasyHSM, a low-maintenance hierarchical storage management solution for IBM Spectrum Scale (GPFS). EasyHSM is a plugin for the GPFS HSM subsystem that supports various backend storage technologies in a cost-effective way without capacity licensing. It integrates fully with GPFS' information lifecycle management policies to automatically tier hot and cold data between high-performance and low-cost storage.

Technology

1
A low maintenance,
highly effective HSM solution
for IBM Spectrum Scale (GPFS)
EasyHSM Overview
Jan 2017 Igor Sfiligoi
Hot Data
Cold Data

2
GPFS HSM at a glance
Hot Data
Automated Tiering to Keep Storage Costs in check
The right data in the right place at the right cost
Cold Data
(GPFS)
HSM = Hierarchical Storage Management
Object Storage
NAS
Open Source Cluster Storage
Enterprise HPC Cluster Storage

3
Advantages of GPFS
GPFS
Clients
(POSIX)
NFSSMB
• High Performance System
• Scale out architecture
• Enterprise class
• Provides industry standard interfaces
(GPFS)
HSM enabled

4
The place of EasyHSM
Hot Data
A HSM-plugin for GPFS HSM subsystem
Supports many backend storage technologies
Cold Data
(GPFS)
EasyHSM

5
Why EasyHSM vs other HSM solutions?
 Easy to use
 Load and go, no need for expensive professional integration services
 Can be loaded on existing GPFS hardware
 Fully integrated with GPFS ILM policy manager
 Affordable
 No capacity-based licensing
Far more affordable than capacity-licensed HSM products
 Works with cost-effective IBM
Spectrum Scale Standard Edition Licenses

6
For more information
Feel free to contact me at
Igor.Sfiligoi@ga.com
Or shoot an mail at the product manager at
Robert.Murphy@ga.com
EasyHSM home page:
http://www.ga.com/easyhsm

Alluxio Online Community Office Hours Jan 28, 2020 Speakers: Bin Fan, Alluxio Calvin Jia, Alluxio Alluxio 2.0 release was the biggest update since the birth of the project “Tachyon” from UC Berkley’s AmpLab. Gathering feedback from our Open Source Community and enterprise users, Alluxio 2.0 expands the system in three major directions including improving the operability of the system, having more advanced data management, as well as re-architecting the system to be able to scale to 1 billion + file. The system is now cloud native on AWS, Google Cloud, and allow users to enable native deployment with K8s. The new advanced data management enables data migration and replication from diff storage systems. In this office hour, we introduce what’s new in the Alluxio 2 release, and dive deeper in each major direction the system has expanded on. In this Office Hour, we will go over: - Introduction and motivation of focus areas of Alluxio 2 - Overview of cloud native deployment methods - New data management features - System scalability improvements

Using Alluxio as a Fault-tolerant Pluggable Optimization Component of JD.com'...

Alluxio, Inc.

Spark Summit EU talk by Jiri Simsa

Spark Summit

Open Source Memory Speed Virtual Distributed Storage

Alluxio, Inc.

Enable Fast Big Data Analytics on Ceph with Alluxio at Ceph Days 2017

Alluxio, Inc.

IMC Summit 2016 Breakout - Brian Bulkowski - NVMe, Storage Class Memory and O...

In-Memory Computing Summit

Much industry focus is on All-Flash Arrays with traditional databases, but new databases using native direct-attached Flash have proven reliable, performant, and popular for operational use cases. Today, these operational databases store account information for banking and retail applications, real-time routing information for telecoms, and user profiles for advertising; they also support machine learning for applications in the financial industry, such as fraud detection. While proprietary PCIe and “wide SATA” had previously been popular, NVMe has finally come into operational use. Aerospike will discuss the benefits of NVMe for these use cases (including specific configurations and performance numbers), as well as the architectural implications of low-latency Flash and Storage Class Memory.

Building Fast SQL Analytics on Anything with Presto, Alluxio

Alluxio, Inc.

Using next gen storage in Cloudstack

ShapeBlue

Everything you wanted to know about RadosGW - Orit Wasserman, Matt Benjamin

Ceph Community

RubiX

Shubham Tagra

Introduction to CloudStack Storage Subsystem

buildacloud

Rise of Intermediate APIs - Beam and Alluxio at Alluxio Meetup 2016

Alluxio, Inc.

Introduction to Alluxio (formerly Tachyon) and how it brings up to 300x perfo...

Alluxio, Inc.

RGW S3: Features vs deep compatibility - Robin Johnson

Ceph Community

Ultra-fast SQL Analytics using PAS (Presto on Alluxio Stack)

Alluxio, Inc.

Hybrid collaborative tiered storage with alluxio

Thai Bui

Systems that deal with AWS S3 often come with a negative performance impact. There's no co-location and the data has to move through slower, often congested wire networks. Alluxio can provide a caching layer for the data, however there's still the question of how and when to move which data. Should all the data by default be cached or should they be cached when used? In this talk, I will explore that gray area in between where the users and the dataset publishers will collaborate to decide what and how the data is cache in a tiered-storage architecture to maximize performance and minimize operating costs.

Augmenting Big Data Analytics with Nirvana

Igor Sfiligoi

Introduction to security in the Open Science Grid - OSG School 2014

Igor Sfiligoi

What's hot

Alluxio (formerly Tachyon): Open Source Memory Speed Virtual Distributed Storage

Alluxio, Inc.

Presto on Alluxio Hands-On Lab

Alluxio, Inc.

Introducing gluster filesystem by aditya

Aditya Chhikara

Alluxio Data Orchestration Platform for the Cloud

Shubham Tagra

What’s new in Alluxio 2: from seamless operations to structured data management

Alluxio, Inc.

Using Alluxio as a Fault-tolerant Pluggable Optimization Component of JD.com'...

Alluxio, Inc.

Spark Summit EU talk by Jiri Simsa

Spark Summit

Open Source Memory Speed Virtual Distributed Storage

Alluxio, Inc.

Enable Fast Big Data Analytics on Ceph with Alluxio at Ceph Days 2017

Alluxio, Inc.

IMC Summit 2016 Breakout - Brian Bulkowski - NVMe, Storage Class Memory and O...

In-Memory Computing Summit

Building Fast SQL Analytics on Anything with Presto, Alluxio

Alluxio, Inc.

Using next gen storage in Cloudstack

ShapeBlue

Everything you wanted to know about RadosGW - Orit Wasserman, Matt Benjamin

Ceph Community

RubiX

Shubham Tagra

Introduction to CloudStack Storage Subsystem

buildacloud

Rise of Intermediate APIs - Beam and Alluxio at Alluxio Meetup 2016

Alluxio, Inc.

Introduction to Alluxio (formerly Tachyon) and how it brings up to 300x perfo...

Alluxio, Inc.

RGW S3: Features vs deep compatibility - Robin Johnson

Ceph Community

Ultra-fast SQL Analytics using PAS (Presto on Alluxio Stack)

Alluxio, Inc.

Hybrid collaborative tiered storage with alluxio

Thai Bui

What's hot (20)

Alluxio (formerly Tachyon): Open Source Memory Speed Virtual Distributed Storage

Presto on Alluxio Hands-On Lab

Introducing gluster filesystem by aditya

Alluxio Data Orchestration Platform for the Cloud

What’s new in Alluxio 2: from seamless operations to structured data management

Using Alluxio as a Fault-tolerant Pluggable Optimization Component of JD.com'...

Spark Summit EU talk by Jiri Simsa

Open Source Memory Speed Virtual Distributed Storage

Enable Fast Big Data Analytics on Ceph with Alluxio at Ceph Days 2017

IMC Summit 2016 Breakout - Brian Bulkowski - NVMe, Storage Class Memory and O...

Building Fast SQL Analytics on Anything with Presto, Alluxio

Using next gen storage in Cloudstack

Everything you wanted to know about RadosGW - Orit Wasserman, Matt Benjamin

RubiX

Introduction to CloudStack Storage Subsystem

Rise of Intermediate APIs - Beam and Alluxio at Alluxio Meetup 2016

Introduction to Alluxio (formerly Tachyon) and how it brings up to 300x perfo...

RGW S3: Features vs deep compatibility - Robin Johnson

Ultra-fast SQL Analytics using PAS (Presto on Alluxio Stack)

Hybrid collaborative tiered storage with alluxio

Viewers also liked

Augmenting Big Data Analytics with Nirvana

Igor Sfiligoi

Introduction to security in the Open Science Grid - OSG School 2014

Igor Sfiligoi

LASCON 2013 - AWS CLoud HSM

Oleg Gryb

Survey of clustered_parallel_file_systems_004_lanl.ppt

Rohn Wood

E-Passport: Deploying Hardware Security Modules to Ensure Data Authenticity a...

SafeNet

Facilitating Collaborative Life Science Research in Commercial & Enterprise E...

Chris Dagdigian

Ncar globally accessible user environment

inside-BigData.com

Quatre experiments de Física amb làser per a 2n de Batxillerat

Marcel Jorba

Architecture of the Upcoming OrangeFS v3 Distributed Parallel File System

All Things Open

Ibm spectrum scale fundamentals workshop for americas part 6 spectrumscale el...

xKinAnx

BioIT World 2016 - HPC Trends from the Trenches

Chris Dagdigian

As presented at BioIT World 2016. In one of the more popular presentations of the Expo, Chris delivers a candid assessment of the best, the worthwhile, and the most overhyped information technologies (IT) for life sciences. He’ll cover what has changed (or not) in the past year around infrastructure, storage, computing, and networks. This presentation will help you understand IT to build and support data intensive science. Video link from the presentation: biote.am/bs [Note: email chris@bioteam.net if you would like a PDF copy of this presentation]

Cortana Analytics Workshop: Azure Data Catalog

MSAdvAnalytics

Julie Strauss. This session introduces the newest services in the Cortana Analytics family. The Azure Data Catalog is an enterprise-wide metadata catalog that enables self-service data source discovery. Data Catalog is a fully managed service that stores, describes, indexes, and provides information on how to access any registered data source in your organization. This session presents an overview of the Data Catalog and how – by using it to register, enrich, discover, understand and consume data sources – you can close the gap between those seeking information and those creating it.

A escolha da profissão!

Marina Sorrentino

Big Lab Problems Solved with Spectrum Scale: Innovations for the Coral Program

inside-BigData.com

Deploying a Governed Data Lake

WaterlineData

Ibm spectrum scale_backup_n_archive_v03_ash

Ashutosh Mate

HSM Basic Training

Md. Budrul Hasan Bhuiyan

Ibm spectrum scale fundamentals workshop for americas part 1 components archi...

xKinAnx

Implementing a Data Lake with Enterprise Grade Data Governance

Hortonworks

Hadoop provides a powerful platform for data science and analytics, where data engineers and data scientists can leverage myriad data from external and internal data sources to uncover new insight. Such power is also presenting a few new challenges. On the one hand, the business wants more and more self-service, and on the other hand IT is trying to keep up with the demand for data, while maintaining architecture and data governance standards. In this webinar, Andrew Ahn, Data Governance Initiative Product Manager at Hortonworks, will address the gaps and offer best practices in providing end-to-end data governance in HDP. Andrew Ahn will be followed by Oliver Claude of Waterline Data, who will share a case study of how Waterline Data Inventory works with HDP in the Modern Data Architecture to automate the discovery of business and compliance metadata, data lineage, as well as data quality metrics.

Viewers also liked (19)

Augmenting Big Data Analytics with Nirvana

Introduction to security in the Open Science Grid - OSG School 2014

LASCON 2013 - AWS CLoud HSM

Survey of clustered_parallel_file_systems_004_lanl.ppt

E-Passport: Deploying Hardware Security Modules to Ensure Data Authenticity a...

Facilitating Collaborative Life Science Research in Commercial & Enterprise E...

Ncar globally accessible user environment

Quatre experiments de Física amb làser per a 2n de Batxillerat

Architecture of the Upcoming OrangeFS v3 Distributed Parallel File System

Ibm spectrum scale fundamentals workshop for americas part 6 spectrumscale el...

BioIT World 2016 - HPC Trends from the Trenches

Cortana Analytics Workshop: Azure Data Catalog

A escolha da profissão!

Big Lab Problems Solved with Spectrum Scale: Innovations for the Coral Program

Deploying a Governed Data Lake

Ibm spectrum scale_backup_n_archive_v03_ash

HSM Basic Training

Ibm spectrum scale fundamentals workshop for americas part 1 components archi...

Implementing a Data Lake with Enterprise Grade Data Governance

Similar to EasyHSM Overview

IBM general parallel file system - introduction

IBM Danmark

Introducing IBM Spectrum Scale 4.2 and Elastic Storage Server 3.5

Doug O'Flaherty

Technical Comuting Solutions Made Simple - ISC13 IBM System x Solution

Intel IT Center

Hadoop and Spark Analytics over Better Storage

Sandeep Patil

IBM Platform Computing Elastic Storage

Patrick Bouillaud

IBM Spectrum Scale ECM - Winning Combination

Sasikanth Eda

Apresentação Storage v3700 IBM® Storwize V3700

Wellington Torrejais da Silva

Gestione gerarchica dei dati con SUSE Enterprise Storage e HPE DMF

SUSE Italy

Spectrum Scale Unified File and Object with WAN Caching

Sandeep Patil

Software Defined Analytics with File and Object Access Plus Geographically Di...

Trishali Nayar

Breaking the Silos: Storage for Analytics & AI

DataWorks Summit

Data analytics, Spark, Hadoop and AI have become fundamental tools to drive digital transformation. A critical challenge is moving from isolated experiments to an organizational or enterprise production infrastructure. In this talk, we break apart the modern data analytics workflow to focus on the data challenges across different phases of the analytics and AI life cycle. By presenting a unified approach to data storage for AI and Analytics, organizations can reduce costs, modernize their data strategy and build a sustainable enterprise data lake. By anticipating how Hadoop, Spark, Tensorflow, Caffe and traditional analytics like SAS, HPC can share data, IT departments and data science practitioners can not only co-exist, but speed time to insight. We will present the tangible benefits of a Reference Architecture using real-world installations that span proprietary and open-source frameworks. Using intelligent software-defined shared storage, users are able to eliminate silos, reduce multiple data copies, and improve time to insight.PALLAVI GALGALI, Offering Manager,IBM and DOUGLAS O'FLAHERTY, Portfolio Product Manager, IBM

XtreemStore - A Scalable Storage Management Software Without Limits

Intel IT Center

Hyper-convergence – The only way to the software-defined data center? - Gerno...

Fujitsu Middle East

A reality-check of hyper converged infrastructures Hyper-converged infrastructures are the rising stars in data centers. What is behind this hype? What are its key benefits and what does it mean in practice? Is hyper-convergence always preferable a classic IT architecture? For which use cases is hyper-converged strongly recommended or even a must? And in which way is hyper-convergence fitting in software-defined data centers? This session provides appropriate answers. Gernot Fels, Head of Integrated Systems, Global Product Marketing, Fujitsu

FlashelasticBruce Lethaby

Multi-tenancy: A Core commercetools Differentiator

Kelly Goetsch

Storage cloud and spectrum update February 2016

Joe Krotz

Fujitsu World Tour 2017: Hyper Converged Data Centre

Fujitsu India

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Alluxio Webinar April 6, 2021 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Cleversafe.PPTXPatrick Berghaeger

#FMS2018 NGD Systems Real World Results with #ComputationalStorage

Scott Shadley, MBA,PMC-III

Similar to EasyHSM Overview (20)

IBM general parallel file system - introduction

Introducing IBM Spectrum Scale 4.2 and Elastic Storage Server 3.5

Technical Comuting Solutions Made Simple - ISC13 IBM System x Solution

Hadoop and Spark Analytics over Better Storage

IBM Platform Computing Elastic Storage

IBM Spectrum Scale ECM - Winning Combination

Apresentação Storage v3700 IBM® Storwize V3700

Gestione gerarchica dei dati con SUSE Enterprise Storage e HPE DMF

Spectrum Scale Unified File and Object with WAN Caching

Software Defined Analytics with File and Object Access Plus Geographically Di...

Breaking the Silos: Storage for Analytics & AI

XtreemStore - A Scalable Storage Management Software Without Limits

Hyper-convergence – The only way to the software-defined data center? - Gerno...

Flashelastic

Multi-tenancy: A Core commercetools Differentiator

Storage cloud and spectrum update February 2016

Fujitsu World Tour 2017: Hyper Converged Data Centre

Accelerate Analytics and ML in the Hybrid Cloud Era

Cleversafe.PPTX

#FMS2018 NGD Systems Real World Results with #ComputationalStorage

More from Igor Sfiligoi

Preparing Fusion codes for Perlmutter - CGYRO

Igor Sfiligoi

O&C Meeting - Evaluation of ARM CPUs for IceCube available through Google Kub...

Igor Sfiligoi

Comparing single-node and multi-node performance of an important fusion HPC c...

Igor Sfiligoi

Fusion simulations have traditionally required the use of leadership scale High Performance Computing (HPC) resources in order to produce advances in physics. The impressive improvements in compute and memory capacity of many-GPU compute nodes are now allowing for some problems that once required a multi-node setup to be also solvable on a single node. When possible, the increased interconnect bandwidth can result in order of magnitude higher science throughput, especially for communication-heavy applications. In this paper we analyze the performance of the fusion simulation tool CGYRO, an Eulerian gyrokinetic turbulence solver designed and optimized for collisional, electromagnetic, multiscale simulation, which is widely used in the fusion research community. Due to the nature of the problem, the application has to work on a large multi-dimensional computational mesh as a whole, requiring frequent exchange of large amounts of data between the compute processes. In particular, we show that the average-scale nl03 benchmark CGYRO simulation can be run at an acceptable speed on a single Google Cloud instance with 16 A100 GPUs, outperforming 8 NERSC Perlmutter Phase1 nodes, 16 ORNL Summit nodes and 256 NERSC Cori nodes. Moving from a multi-node to a single-node GPU setup we get comparable simulation times using less than half the number of GPUs. Larger benchmark problems, however, still require a multi-node HPC setup due to GPU memory capacity needs, since at the time of writing no vendor offers nodes with a sufficient GPU memory setup. The upcoming external NVSWITCH does however promise to deliver an almost equivalent solution for up to 256 NVIDIA GPUs. Presented at PEARC22. Paper DOI: https://doi.org/10.1145/3491418.3535130

The anachronism of whole-GPU accounting

Igor Sfiligoi

NVIDIA has been making steady progress in increasing the compute performance of its GPUs, resulting in order of magnitude compute throughput improvements over the years. With several models of GPUs coexisting in many deployments, the traditional accounting method of treating all GPUs as being equal is not reflecting compute output anymore. Moreover, for applications that require significant CPU-based compute to complement the GPU-based compute, it is becoming harder and harder to make full use of the newer GPUs, requiring sharing of those GPUs between multiple applications in order to maximize the achievable science output. This further reduces the value of whole-GPU accounting, especially when the sharing is done at the infrastructure level. We thus argue that GPU accounting for throughput-oriented infrastructures should be expressed in GPU core hours, much like it is normally done for the CPUs. While GPU core compute throughput does change between GPU generations, the variability is similar to what we expect to see among CPU cores. To validate our position, we present an extensive set of run time measurements of two IceCube photon propagation workflows on 14 GPU models, using both on-prem and Cloud resources. The measurements also outline the influence of GPU sharing at both HTCondor and Kubernetes infrastructure level. Presented at PEARC22. Document DOI: https://doi.org/10.1145/3491418.3535125

Auto-scaling HTCondor pools using Kubernetes compute resources

Igor Sfiligoi

HTCondor has been very successful in managing globally distributed, pleasantly parallel scientific workloads, especially as part of the Open Science Grid. HTCondor system design makes it ideal for integrating compute resources provisioned from anywhere, but it has very limited native support for autonomously provisioning resources managed by other solutions. This work presents a solution that allows for autonomous, demand-driven provisioning of Kubernetes-managed resources. A high-level overview of the employed architectures is presented, paired with the description of the setups used in both on-prem and Cloud deployments in support of several Open Science Grid communities. The experience suggests that the described solution should be generally suitable for contributing Kubernetes-based resources to existing HTCondor pools. Presented at PEARC22. Paper DOI: https://doi.org/10.1145/3491418.3535123

Speeding up bowtie2 by improving cache-hit rate

Igor Sfiligoi

Performance Optimization of CGYRO for Multiscale Turbulence Simulations

Igor Sfiligoi

$Comparing GPU effectiveness for Unifrac distance compute$ $Comparing GPU effectiveness for Unifrac distance compute$

Comparing GPU effectiveness for Unifrac distance compute

Igor Sfiligoi

Managing Cloud networking costs for data-intensive applications by provisioni...

Igor Sfiligoi

Presented at PEARC21. Many scientific high-throughput applications can benefit from the elastic nature of Cloud resources, especially when there is a need to reduce time to completion. Cost considerations are usually a major issue in such endeavors, with networking often a major component; for data-intensive applications, egress networking costs can exceed the compute costs. Dedicated network links provide a way to lower the networking costs, but they do add complexity. In this paper we provide a description of a 100 fp32 PFLOPS Cloud burst in support of IceCube production compute, that used Internet2 Cloud Connect service to provision several logically-dedicated network links from the three major Cloud providers, namely Amazon Web Services, Microsoft Azure and Google Cloud Platform, that in aggregate enabled approximately 100 Gbps egress capability to on-prem storage. It provides technical details about the provisioning process, the benefits and limitations of such a setup and an analysis of the costs incurred.

Accelerating Key Bioinformatics Tasks 100-fold by Improving Memory Access

Igor Sfiligoi

Presented at PEARC21. Most experimental sciences now rely on computing, and biolog- ical sciences are no exception. As datasets get bigger, so do the computing costs, making proper optimization of the codes used by scientists increasingly important. Many of the codes developed in recent years are based on the Python-based NumPy, due to its ease of use and good performance characteristics. The composable nature of NumPy, however, does not generally play well with the multi-tier nature of modern CPUs, making any non-trivial multi- step algorithm limited by the external memory access speeds, which are hundreds of times slower than the CPU’s compute capabilities. In order to fully utilize the CPU compute capabilities, one must keep the working memory footprint small enough to fit in the CPU caches, which requires splitting the problem into smaller portions and fusing together as many steps as possible. In this paper, we present changes based on these principles to two important func- tions in the scikit-bio library, principal coordinates analysis and the Mantel test, that resulted in over 100x speed improvement in these widely used, general-purpose tools.

Using A100 MIG to Scale Astronomy Scientific Output

Igor Sfiligoi

Presented at GTC21. The raw computing power of GPUs has been steadily increasing, significantly outpacing the CPU gains. This poses a problem for many GPU-enabled scientific applications that use CPU code paths to feed data to the GPU code, resulting in lower GPU utilization, and thus reduced gains in scientific output. Applications that are high-throughput in nature, such as astronomy-focused IceCube and LIGO, can partially work around the problem by running several instances of the executable on the same GPU. This approach, however, is sub-optimal both in terms of application performance and workflow management complexity. The recently introduced Multi-Instance GPU (MIG) capability, available on the NVIDIA A100 GPU, provides a much cleaner and easier-to-use alternative by allowing the logical slicing of the powerful GPU and assigning different slices to different applications. And at least in the case of IceCube, it can provide over 3x more scientific output on the same hardware.

Using commercial Clouds to process IceCube jobs

Igor Sfiligoi

Presented at EDUCAUSE CCCG March 2021. The IceCube Neutrino Observatory is the world’s premier facility to detect neutrinos. Built at the south pole in natural ice, it requires extensive and expensive calibration to properly track the neutrinos. Most of the required compute power comes from on-prem resources through the Open Science Grid, but IceCube can easily harness the Cloud compute at any scale, too, as demonstrated by a series of Cloud bursts. This talk provides both details of the performed Cloud bursts, as well as some insight in the science itself.

Modest scale HPC on Azure using CGYRO

Igor Sfiligoi

Fusion simulations have traditionally required the use of leadership scale HPC resources in order to produce advances in physics. One such package is CGYRO, a premier tool for multi-scale plasma turbulence simulation. CGYRO is a typical HPC application that will not fit into a single node, as it requires several TeraBytes of memory and O(100) TFLOPS compute capability for cutting-edge simulations. CGYRO also requires high-throughput and low-latency networking, due to its reliance on global FFT computations. While in the past such compute may have required hundreds, or even thousands of nodes, recent advances in hardware capabilities allow for just tens of nodes to deliver the necessary compute power. We explored the feasibility of running CGYRO on Cloud resources provided by Microsoft on their Azure platform, using the infiniband-connected HPC resources in spot mode. We observed both that CPU-only resources were very efficient, and that running in spot mode was doable, with minimal side effects. The GPU-enabled resources were less cost effective but allowed for higher scaling.

Data-intensive IceCube Cloud Burst

Igor Sfiligoi

Scheduling a Kubernetes Federation with Admiralty

Igor Sfiligoi

Accelerating microbiome research with OpenACC

Igor Sfiligoi

Presented at OpenACC Summit 2020. UniFrac is a commonly used metric in microbiome research for comparing microbiome profiles to one another. Computing UniFrac on modest sample sizes used to take a workday on a server class CPU-only node, while modern datasets would require a large compute cluster to be feasible. After porting to GPUs using OpenACC, the compute of the same modest sample size now takes only a few minutes on a single NVIDIA V100 GPU, while modern datasets can be processed on a single GPU in hours. The OpenACC programming model made the porting of the code to GPUs extremely simple; the first prototype was completed in just over a day. Getting full performance did however take much longer, since proper memory access is fundamental for this application.

Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...

Igor Sfiligoi

Presented at PEARC20. This talk presents expanding the IceCube’s production HTCondor pool using cost-effective GPU instances in preemptible mode gathered from the three major Cloud providers, namely Amazon Web Services, Microsoft Azure and the Google Cloud Platform. Using this setup, we sustained for a whole workday about 15k GPUs, corresponding to around 170 PFLOP32s, integrating over one EFLOP32 hour worth of science output for a price tag of about $60k. In this paper, we provide the reasoning behind Cloud instance selection, a description of the setup and an analysis of the provisioned resources, as well as a short description of the actual science output of the exercise.

Porting and optimizing UniFrac for GPUs

Igor Sfiligoi

Poster presented at PEARC20. UniFrac is a commonly used metric in microbiome research for comparing microbiome profiles to one another (“beta diversity”). The recently implemented Striped UniFrac added the capability to split the problem into many independent subproblems and exhibits near linear scaling. In this poster we describe steps undertaken in porting and optimizing Striped Unifrac to GPUs. We reduced the run time of computing UniFrac on the published Earth Microbiome Project dataset from 13 hours on an Intel Xeon E5-2680 v4 CPU to 12 minutes on an NVIDIA Tesla V100 GPU, and to about one hour on a laptop with NVIDIA GTX 1050 (with minor loss in precision). Computing UniFrac on a larger dataset containing 113k samples reduced the run time from over one month on the CPU to less than 2 hours on the V100 and 9 hours on an NVIDIA RTX 2080TI GPU (with minor loss in precision). This was achieved by using OpenACC for generating the GPU offload code and by improving the memory access patterns. A BSD-licensed implementation is available, which produces a Cshared library linkable by any programming language.

Demonstrating 100 Gbps in and out of the public Clouds

Igor Sfiligoi

Poster presented at PEARC20. There is increased awareness and recognition that public Cloud providers do provide capabilities not found elsewhere, with elasticity being a major driver. The value of elastic scaling is however tightly coupled to the capabilities of the networks that connect all involved resources, both in the public Clouds and at the various research institutions. This poster presents results of measurements involving file transfers inside public Cloud providers, fetching data from on-prem resources into public Cloud instances and fetching data from public Cloud storage into on-prem nodes. The networking of the three major Cloud providers, namely Amazon Web Services, Microsoft Azure and the Google Cloud Platform, has been benchmarked. The on-prem nodes were managed by either the Pacific Research Platform or located at the University of Wisconsin – Madison. The observed sustained throughput was of the order of 100 Gbps in all the tests moving data in and out of the public Clouds and throughput reaching into the Tbps range for data movements inside the public Cloud providers themselves. All the tests used HTTP as the transfer protocol.

TransAtlantic Networking using Cloud links

Igor Sfiligoi

More from Igor Sfiligoi (20)

Preparing Fusion codes for Perlmutter - CGYRO

O&C Meeting - Evaluation of ARM CPUs for IceCube available through Google Kub...

Comparing single-node and multi-node performance of an important fusion HPC c...

The anachronism of whole-GPU accounting

Auto-scaling HTCondor pools using Kubernetes compute resources

Speeding up bowtie2 by improving cache-hit rate

Performance Optimization of CGYRO for Multiscale Turbulence Simulations

$Comparing GPU effectiveness for Unifrac distance compute$ $Comparing GPU effectiveness for Unifrac distance compute$

Comparing GPU effectiveness for Unifrac distance compute

Managing Cloud networking costs for data-intensive applications by provisioni...

Accelerating Key Bioinformatics Tasks 100-fold by Improving Memory Access

Using A100 MIG to Scale Astronomy Scientific Output

Using commercial Clouds to process IceCube jobs

Modest scale HPC on Azure using CGYRO

Data-intensive IceCube Cloud Burst

Scheduling a Kubernetes Federation with Admiralty

Accelerating microbiome research with OpenACC

Demonstrating a Pre-Exascale, Cost-Effective Multi-Cloud Environment for Scie...

Porting and optimizing UniFrac for GPUs

Demonstrating 100 Gbps in and out of the public Clouds

TransAtlantic Networking using Cloud links

Recently uploaded

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

FIDO Alliance

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

Bits & Pixels using AI for Good.........

Alison B. Lowndes

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

When stars align: studies in data quality, knowledge graphs, and machine lear...

Elena Simperl

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Sri Ambati

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Paige Cruz

Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack. While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack. I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:

Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx

nkrafacyberclub

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

By Design, not by Accident - Agile Venture Bolzano 2024

Pierluigi Pugliese

Quantum Computing: Current Landscape and the Future Role of APIs

Vlad Stirbu

Free Complete Python - A step towards Data Science

RinaMondal9

The Future of Platform Engineering

Jemma Hussein Allen

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

Assure Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Product School

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

Introduction to CHERI technology - Cybersecurity

mikeeftimakis1

UiPath Test Automation using UiPath Test Suite series, part 4

DianaGray10

Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap. The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies. Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques What will you get from this session? 1. Insights into SAP testing best practices 2. Heatmap utilization for testing 3. Optimization of testing processes 4. Demo Topics covered: Execution from the test manager Orchestrator execution result Defect reporting SAP heatmap example with demo Speaker: Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

Recently uploaded (20)

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

Essentials of Automations: Optimizing FME Workflows with Parameters

Bits & Pixels using AI for Good.........

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

When stars align: studies in data quality, knowledge graphs, and machine lear...

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Secstrike : Reverse Engineering & Pwnable tools for CTF.pptx

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

By Design, not by Accident - Agile Venture Bolzano 2024

Quantum Computing: Current Landscape and the Future Role of APIs

Free Complete Python - A step towards Data Science

The Future of Platform Engineering

PCI PIN Basics Webinar from the Controlcase Team

Assure Contact Center Experiences for Your Customers With ThousandEyes

De-mystifying Zero to One: Design Informed Techniques for Greenfield Innovati...

Leading Change strategies and insights for effective change management pdf 1.pdf

Introduction to CHERI technology - Cybersecurity

UiPath Test Automation using UiPath Test Suite series, part 4

EasyHSM Overview

1. 1 A low maintenance, highly effective HSM solution for IBM Spectrum Scale (GPFS) EasyHSM Overview Jan 2017 Igor Sfiligoi Hot Data Cold Data

2. 2 GPFS HSM at a glance Hot Data Automated Tiering to Keep Storage Costs in check The right data in the right place at the right cost Cold Data (GPFS) HSM = Hierarchical Storage Management Object Storage NAS Open Source Cluster Storage Enterprise HPC Cluster Storage

3. 3 Advantages of GPFS GPFS Clients (POSIX) NFSSMB • High Performance System • Scale out architecture • Enterprise class • Provides industry standard interfaces (GPFS) HSM enabled

4. 4 The place of EasyHSM Hot Data A HSM-plugin for GPFS HSM subsystem Supports many backend storage technologies Cold Data (GPFS) EasyHSM

5. 5 Why EasyHSM vs other HSM solutions?  Easy to use  Load and go, no need for expensive professional integration services  Can be loaded on existing GPFS hardware  Fully integrated with GPFS ILM policy manager  Affordable  No capacity-based licensing Far more affordable than capacity-licensed HSM products  Works with cost-effective IBM Spectrum Scale Standard Edition Licenses

6. 6 For more information Feel free to contact me at Igor.Sfiligoi@ga.com Or shoot an mail at the product manager at Robert.Murphy@ga.com EasyHSM home page: http://www.ga.com/easyhsm

EasyHSM Overview

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (19)

Similar to EasyHSM Overview

Similar to EasyHSM Overview (20)

More from Igor Sfiligoi

More from Igor Sfiligoi (20)

Recently uploaded

Recently uploaded (20)

EasyHSM Overview