Optimizing Lustre and GPFS with DDN

•

3 likes•3,653 views

This document discusses DDN's optimization of Lustre and GPFS file systems. It provides an overview of DDN's extensive testing and benchmarking facilities and describes their long involvement with the Lustre file system, including major contributions to the open source code. It also presents performance results demonstrating the benefits of various DDN technologies and configurations.

Technology

Optimizing Lustre and GPFS
Solutions with DDN
Robert Triendl
VP of Worldwide HPC Strategy,
DataDirect Networks

2!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
File Systems @ DDN

3!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
File System Basics
•  File system are where your data lives
•  File systems are complex software level
technologies…
•  … so there are always surprises!
•  There are huge differences in performance,
functionality, and reliability
•  When it comes to performance, no file system
fits all requirements

4!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
Test and Benchmark Labs

5!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
GTLS | Benchmark Lab Sites
EMEA Lab
Dusseldorf, Germany
Asia Pacific Lab
Tokyo, Japan
East Coast Lab
Columbia, MD
West Coast Lab
Sunnyvale, CA

6!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
DDN and Lustre
•  Started with Lustre 0.6, and the first
commercial Lustre support contract with CFS!
•  Over 250 EXAScaler customers worldwide
today and many more using DDN storage for
Lustre
•  Customers in many industries (HPC centers,
Large Experimental Facilities, Oil & Gas, Life
Science, Automotive, etc.)
•  Very broad set of applications supported

7!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
Corp
Data
4%
Government
Security 17%
Research
Data
Analysis,
28%HPC
Archive
18%
HPC
Work
20%
HPC Work
Corp 12%
Project Quota
Metadata Perf
SSD Acceleration
Fine-Grained Monitoring
NFS/CIFS Access
Management
Connectors
Object/Cloud Links
Data Management
Backup/Replication
HSM
Client Performance
Cluster Integration
Large I/O
IME Caching
Security Features
Lustre WAN
RAS
Small File I/O

8!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
DDN Open Source Lustre Contributions
0
20
40
60
80
100
120
140
160
180
2.1 2.4.0 2.3.50-2.4.0 2.5.0 2.5.50-2.6.0
EMC
CEA
SUSE
Bull
Other
Cray
LLNN
Xyratex
DDN

9!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
Large RPC Size Effects
0%!
20%!
40%!
60%!
80%!
100%!
120%!
0! 100! 200! 300! 400! 500! 600! 700!
Number of Process!
WRITE!
7.2KSAS(1MB RPC)! 7.2KSAS(4MB RPC)! SSD(1MB RPC)!
0%!
20%!
40%!
60%!
80%!
100%!
120%!
0! 100! 200! 300! 400! 500! 600! 700!
Number of Process!
READ!
7.2KSAS(1MB RPC)! 7.2KSAS(4MB RPC)! SSD(1MB RPC)!

10!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
•  Limited
single
client
scaling

•  Good
scaling

with
clock

speed

•  Good
Scaling

with
core

count
and
HT

•  Great
Scaling
with
DNE

•  Limita<ons
on
Dir
Creates
(TBD)

Lustre
Metadata

11!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
mmap() I/O
Performance Improvements
0!
100!
200!
300!
400!
500!
lustre-1.8.9! lustre-2.5.2! DDN branch!
mmap() Read Performance !
(1MB block size)!
0!
100!
200!
300!
400!
500!
32K! 128K! 512K! 1024K!
mmap() Read Performance!
Lustre-1.8.9! DDN branch!

12!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
EXAScaler Monitoring
OSS/MDS!
collectd!
Lustre
client!
DDN
monitoring

plugin!
graphite!
Monitoring
Server!
collectd!
Graphite
plugin!
UDP(TCP)/IP

based
small

text
message

transfer
graphite!
•  Lightweight

•  Near
real-‐<me

•  Massive
scale

•  Customizable

•  File system, OST Pool, OST/MDT stats, etc.
•  JOB ID, UID/GID, aggregation of application's
stats, etc.
•  Archive of data by policy

13!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
EXAScaler Monitoring
•  Running in TITECH
–  over 112 Object Storage
Targets across
–  1700 clients
•  That’s around 1M
statistics
•  Need to store every few
seconds
•  Demo of over 10M stats
at DDN Booth

14!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
VMs on GRIDScaler
256 VMs on 16 Clients
0!
2000!
4000!
6000!
8000!
10000!
12000!
14000!
16000!
1! 2! 4! 8! 16! 32! 64! 128! 256!
Throughput(MB/sec)!
Number of Process!
0!
2000!
4000!
6000!
8000!
10000!
12000!
1! 2! 4! 8! 16! 32! 64! 128! 256!
Throughput(MB/sec)!
Number of Process!

15!
© 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others.
Any statements or representations around future events are subject to change. ddn.com
0!
100!
200!
300!
400!
500!
600!
700!
800!
900!
1000!
1! 10! 20! 30! 40!
Total Bandwidth!
Read Bandwidth! Write Bandwidth!
GRIDScaler for OpenStack
vbench Results
0!
1000!
2000!
3000!
4000!
5000!
6000!
1! 10! 20! 30! 40!
Total IOPS!
Read IOPS! Write IOPS!

What's hot

Jean Thomas Acquaviva from DDN present this deck at the 2016 HPC Advisory Council Switzerland Conference. "Thanks to the arrival of SSDs, the performance of storage systems can be boosted by orders of magnitude. While a considerable amount of software engineering has been invested in the past to circumvent the limitations of rotating media, there is a misbelief than a lightweight software approach may be sufficient for taking advantage of solid state media. Taking the data protection as an example, this talk will present some of the limitations of current storage software stacks. We will then discuss how this unfold to a more radical re-design of the software architecture and ultimately is making a case for an I/O interception layer." Learn more: http://ddn.com Watch the video presentation: http://wp.me/p3RLHQ-f7J Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

DDN: Protecting Your Data, Protecting Your Hardware

inside-BigData.com

Today, data drives discovery. And discoveries create are key to creating sustained advantages. The better your critical workflows are able to create and access data – the better you’ll be able to discover new, innovative solutions to important problems, or to create entirely new products. More than ever before, data intensive applications need the sustained performance and virtually unlimited scalability that only parallel storage software delivers. Designed for maximum performance and scale, storage solutions powered by Lustre software deliver the performance at scale to meet today’s storage requirements. As the most widely used parallel storage system for HPC, Lustre-powered storage is the ideal storage foundation. But scalable performance storage by itself only solves half the problem. Today’s users expect storage solutions that deliver sustained performance, scale upward to near limitless capacities, and are simple to install and manage. Intel(r) Enterprise Edition for Lustre* software combines the straight line speed and scale of Lustre with the bottom line need for lowered management complexity and cost. As the recognized leaders in the development and support of the Lustre file system, Intel has the expertise to make storage solutions for data intensive applications faster, smarter and easier.

The Importance of Fast, Scalable Storage for Today’s HPC

Intel IT Center

Blazing Fast Lustre Storage

Intel IT Center

Dell Lustre Storage Architecture Presentation - MBUG 2016

Andrew Underwood

32992 lam ebc storage overview3

gmazuel

Performance Comparison of Intel Enterprise Edition Lustre and HDFS for MapRed...

inside-BigData.com

Cisco NetApp VMware - Long Distance VMotion

NetApp Virtualization Team

Data Domain Architecture

koesteruk22

DDN: Massively-Scalable Platforms and Solutions Engineered for the Big Data a...

inside-BigData.com

A key reason for using dynamic tiering for mainframe storage is performance. This session will focus on dynamic tiering in mainframe environments and how to configure and control tiering. The session ends with a detailed discussion of performance considerations when using Hitachi Dynamic Tiering. By viewing this webcast, you will: Understand Hitachi Dynamic Tiering and the options for configuring and controlling tiering. Understand the performance considerations and the type of performance improvements you might experience when you implement Hitachi Dynamic Tiering. For more information on Hitachi Dynamic Tiering please visit: http://www.hds.com/products/storage-software/hitachi-dynamic-tiering.html?WT.ac=us_mg_pro_dyntir

VSP Mainframe Dynamic Tiering Performance Considerations

Hitachi Vantara

Next Generation Data Protection Architecture

Gina Tragos

Hitachi Unified Storage and Hitachi NAS Platform Performance Optimization wit...

Hitachi Vantara

Trends in Data Protection with DCIG

Gina Tragos

CLOSE As enterprises embrace software defined and hyperconverged infrastructure, original methods for defining infrastructure ingredients becomes more complex. Maintaining a balanced platform with a diverse set of workloads is required to maximize TCO. Defining configurations at the solution level helps ease the challenges of implementing HCI, while optimizing TCO. Come to this session to learn about Intel’s view on how HCI configurations will be using technologies like Optane SSDs, the newest server platforms, and new SSD form factors to continue HCI TCO scaling.

Deep Dive On Intel Optane SSDs And New Server Platforms

NEXTtour

Macroview Netapp Overview

Alex Tsui

HPE Solutions for Challenges in AI and Big Data

Lviv Startup Club

HPC DAY 2017 | HPE Storage and Data Management for Big Data

HPC DAY

S cv3179 spectrum-integration-openstack-edge2015-v5

Tony Pearson

EMC config Hadoop

solarisyougood

File And Content Services

HunterFarmer

What's hot (20)

DDN: Protecting Your Data, Protecting Your Hardware

The Importance of Fast, Scalable Storage for Today’s HPC

Blazing Fast Lustre Storage

Dell Lustre Storage Architecture Presentation - MBUG 2016

32992 lam ebc storage overview3

Performance Comparison of Intel Enterprise Edition Lustre and HDFS for MapRed...

Cisco NetApp VMware - Long Distance VMotion

Data Domain Architecture

DDN: Massively-Scalable Platforms and Solutions Engineered for the Big Data a...

VSP Mainframe Dynamic Tiering Performance Considerations

Next Generation Data Protection Architecture

Hitachi Unified Storage and Hitachi NAS Platform Performance Optimization wit...

Trends in Data Protection with DCIG

Deep Dive On Intel Optane SSDs And New Server Platforms

Macroview Netapp Overview

HPE Solutions for Challenges in AI and Big Data

HPC DAY 2017 | HPE Storage and Data Management for Big Data

S cv3179 spectrum-integration-openstack-edge2015-v5

EMC config Hadoop

File And Content Services

Similar to Optimizing Lustre and GPFS with DDN

Academic Workflows with iRODS FINAL

Randy Splinter

DDN Strategic Vision Tour June 2015

inside-BigData.com

HDFS is well designed to operate efficiently at scale for normal hardware failures within a datacenter, but it is not designed to handle significant negative events, such as datacenter failures. To overcome this defect, a common practice of HDFS disaster recovery (DR) is replicating data from one location to another through DistCp, which provides a robust and reliable backup capability for HDFS data through batch operations. However, DistCp also has several drawbacks: (1) Taking HDFS Snapshots is time and space consuming on large HDFS cluster. (2) Applying file changes though MapReduce may introduce additional execution overhead and potential issues. (3) DistCp requires administrator intervene to trigger, perform, and verify DistCp jobs, which is not user-friendly in practice. In this presentation, we will share our experience in HDFS DR and introduce our light-weighted HDFS disaster recovery system that addresses afore-mentioned problems. Different from DistCp, our light-weighted DR system is designed based on HDFS logs (e.g. edit log and Inotify), light-weighted producer/consumer framework, and FileSystem API. During synchronization, it fetches limited subsets of namespace and incremental file changes from NameNode, then our executors apply these changes incrementally to remote clusters through FileSystem API. Furthermore, it also provides a powerful user interface with trigger conditions, path filters and jobs scheduler, etc. Compared to DistCp, it is more straightforward, light-weighted, reliable, efficient, and user-friendly. Speaker Qiyuan Gong, Big Data Software Engineer, Intel

Light-weighted HDFS disaster recovery

DataWorks Summit

Evolution of Big Data at Intel - Crawl, Walk and Run Approach

DataWorks Summit

Introduction to DDS

Rick Warren

The Future of Data Management: The Enterprise Data Hub

Cloudera, Inc.

Verizon – Global Technology Services (GTS) was challenged by a multi-tier, labor-intensive process when trying to migrate data from disparate sources into a data lake to create financial reports and business insights. Join this session to learn more about how Verizon: • Easily accessed data from multiple sources including SAP data • Ingested data into major targets including Hadoop • Achieved real-time insights from data leveraging change data capture (CDC) technology • Reduced costs and labor

Verizon Centralizes Data into a Data Lake in Real Time for Analytics

DataWorks Summit

What’s New in Documentum 7.3

Michael Mohen

How can you make sense of messy data? How do you wrap structure around non-relational, flexibly structured data? With the growth in cloud technologies, how do you balance the need for flexibility and scale with the need for structure and analytics? Join us for an overview of the marketplace today and a review of the tools needed to get the job done. During this hour, we'll cover: - How big data is challenging the limits of traditional data management tools - How to recognize when tools like MongoDB, Hadoop, IBM Cloudant, R Studio, IBM dashDB, CouchDB, and others are the right tools for the job.

Integrating Structure and Analytics with Unstructured Data

DATAVERSITY

Webinar: Cloud Data Masking - Tips to Test Software Securely

Skytap Cloud

At StampedeCon 2014, Jean-Luc Chatelain (DDN) presented 'Big Data: Infrastructure Implications for “The Enterprise of Things”.' The amount of data in our world has been exploding, and storing and analyzing large data sets—so-called big data—will become a key basis of competition for the new “Enterprise of Things”, underpinning fresh waves of productivity growth, innovation, and consumer surplus. Leaders in every sector – from government to healthcare to finance – will have to grapple with the implications of big data, as data growth continues unabated for the foreseeable future. The quest to make sense of all this big data begins with breaking down data silos within organizations using the cost appropriate, shared infrastructure to ensure optimal extraction and analysis of data, knowledge and insight. As the leading global e-commerce service, PayPal has transformed the way the company leverages big data storage and hyper-scale analytics to help improve both the safety and purchasing experiences of its online customers. In this discussion, using real-world customer examples such as PayPal, we will explore what Big Data Storage is from high performance file sharing to long-term archiving, as well as ways to break down data silos to reduce the cost and storage complexity of managing demanding workflows and data environments. We will demonstrate how hyperscale storage can enable near-real-time, stream analytics processing for behavioral and situational modeling, as well as for fraud detection, marketing and systems intelligence. We will ask what the greatest barriers to effective business analytics are and how today’s data analytics platforms, including Hadoop, Vertica, Python and Java, can be optimized to enable machine learning, event streaming, forecasting, and reduce overhead associated with human intervention. You’ll come away from this session understanding the infrastructure implications and options for organizations looking to maximize their big data for competitive advantage.

Big Data: Infrastructure Implications for “The Enterprise of Things” - Stampe...

StampedeCon

Government and Education Webinar: Improving Application Performance

SolarWinds

The blessing and curse of today's database market? So many choices! While relational databases still dominate the day-to-day business, a host of alternatives has evolved around very specific use cases: graph, document, NoSQL, hybrid (HTAP), column store, the list goes on. And the database tools market is teeming with activity as well. Register for this special Research Webcast to hear Dr. Robin Bloor share his early findings about the evolving database market. He'll be joined by Steve Sarsfield of HPE Vertica, and Robert Reeves of Datical in a roundtable discussion with Bloor Group CEO Eric Kavanagh. Send any questions to info@insideanalysis.com, or tweet with #DBSurvival.

Horses for Courses: Database Roundtable

Eric Kavanagh

Cloud is turbocharging the Enterprise IT landscape with agility and flexibility. And now, discussions of cloud architecture dominate Enterprise IT. Cloud is enabling many ephemeral on-demand use cases which is a game-changing opportunity for analytic workloads. But all of this comes with the challenges of running enterprise workloads in the cloud securely and with ease. With the convergence of cloud, IoT, and big data technologies, enterprises increasingly have their data spread across multiple data lakes on-prem and in cloud data lake stores in many geographies and across multiple public cloud vendor platforms, for example, due to regulatory and compliance mandates that limit cross-border data transfer. With the proliferation of data types and sources in this complex landscape, the process of discovery, provisioning, and running relevant workloads on this data to gather insights has become more complex. Additionally, gaining global visibility into the business context, usage, and trustworthiness of data requires a centralized view of all data and metadata, security controls, data access, and monitoring. All of these challenges create a significant chasm between initial data capture and subsequent data insights generation to drive value creation. Therefore, enterprises now require a “global insight fabric” that can find a happy medium between adequate rules and policies of data governance while providing a trusted environment for users to collaborate and share data responsibly in order to create value. In this talk, we will outline how Hortonworks DataPlane Service(DPS) can help customers build a global insight fabric that can span storing and analyzing data within data centers to implementing an open source hybrid architecture that takes advantage of cloud's elasticity and new use cases. We will get a personal view of the challenges faced in safely moving data from on-premises data centers into multiple public clouds, safeguarding it through replication, and then applying consistent security and governance policies across diverse environments to deliver trusted data and insights to the business. We will highlight how DataPlane Service can help enterprises with this hybrid architectural journey, and how open source architectures are enabling this transformation across enterprises.

Running Enterprise Workloads with an open source Hybrid Cloud Data Architecture

DataWorks Summit

Ron Kasabian - Intel Big Data & Cloud Summit 2013

IntelAPAC

Alluxio Tech Talk Jul 17, 2019 Speakers: Brien Porter, Intel Alex Ma, Alluxio The ever increasing challenge to process and extract value from exploding data with AI and analytics workloads makes a memory centric architecture with disaggregated storage and compute more attractive. This decoupled architecture enables users to innovate faster and scale on-demand. Enterprises are also increasingly looking towards object stores to power their big data & machine learning workloads in a cost-effective way. However, object stores don’t provide big data compatible APIs as well as the required performance. In this webinar, the Intel and Alluxio teams will present a proposed reference architecture using Alluxio as the in-memory accelerator for object stores to enable modern analytical workloads such as Spark, Presto, Tensorflow, and Hive. We will also present a technical overview of Alluxio.

Accelerate and Scale Big Data Analytics with Disaggregated Compute and Storage

Alluxio, Inc.

Cloudera 助力台灣大數據產業的發展

Etu Solution

Standardizing the Data Distribution Service (DDS) API for Modern C++

Sumant Tambe

In this deck, James Coomer from DDN presents: Infinite Memory Engine IME. "DDN’s Infinite Memory Engine (IME) is a scale-out, flash-native, software-defined, storage cache that streamlines the data path for application IO. IME interfaces directly to applications and secures IO via a data path that eliminates file system bottlenecks. With IME, architects can realize true flash-cache economics with a storage architecture that separates capacity from performance." Watch the video: https://wp.me/p3RLHQ-hoy Learn more: http://ddn.com Sign up for our insideHPC Newsletter: http://insideHPC.com/newsletter

Infinite Memory Engine: HPC in the FLASH Era

inside-BigData.com

Toward Scalable and Powerful CloudStack

Takashi Kanai

Similar to Optimizing Lustre and GPFS with DDN (20)

Academic Workflows with iRODS FINAL

DDN Strategic Vision Tour June 2015

Light-weighted HDFS disaster recovery

Evolution of Big Data at Intel - Crawl, Walk and Run Approach

Introduction to DDS

The Future of Data Management: The Enterprise Data Hub

Verizon Centralizes Data into a Data Lake in Real Time for Analytics

What’s New in Documentum 7.3

Integrating Structure and Analytics with Unstructured Data

Webinar: Cloud Data Masking - Tips to Test Software Securely

Big Data: Infrastructure Implications for “The Enterprise of Things” - Stampe...

Government and Education Webinar: Improving Application Performance

Horses for Courses: Database Roundtable

Running Enterprise Workloads with an open source Hybrid Cloud Data Architecture

Ron Kasabian - Intel Big Data & Cloud Summit 2013

Accelerate and Scale Big Data Analytics with Disaggregated Compute and Storage

Cloudera 助力台灣大數據產業的發展

Standardizing the Data Distribution Service (DDS) API for Modern C++

Infinite Memory Engine: HPC in the FLASH Era

Toward Scalable and Powerful CloudStack

More from inside-BigData.com

In this deck from the Stanford HPC Conference, Shahin Khan from OrionX describes major market Shifts in IT. "We will discuss the digital infrastructure of the future enterprise and the state of these trends." "We work with clients on the impact of Digital Transformation (DX) on them, their customers, and their messages. Generally, they want to track, in one place, trends like IoT, 5G, AI, Blockchain, and Quantum Computing. And they want to know what these trends mean, how they affect each other, and when they demand action, and how to formulate and execute an effective plan. If that describes you, we can help." Watch the video: https://wp.me/p3RLHQ-lPP Learn more: http://orionx.net and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Major Market Shifts in IT

inside-BigData.com

In this deck from IWOCL / SYCLcon 2020, Hal Finkel from Argonne National Laboratory presents: Preparing to program Aurora at Exascale - Early experiences and future directions. "Argonne National Laboratory’s Leadership Computing Facility will be home to Aurora, our first exascale supercomputer. Aurora promises to take scientific computing to a whole new level, and scientists and engineers from many different fields will take advantage of Aurora’s unprecedented computational capabilities to push the boundaries of human knowledge. In addition, Aurora’s support for advanced machine-learning and big-data computations will enable scientific workflows incorporating these techniques along with traditional HPC algorithms. Programming the state-of-the-art hardware in Aurora will be accomplished using state-of-the-art programming models. Some of these models, such as OpenMP, are long-established in the HPC ecosystem. Other models, such as Intel’s oneAPI, based on SYCL, are relatively-new models constructed with the benefit of significant experience. Many applications will not use these models directly, but rather, will use C++ abstraction libraries such as Kokkos or RAJA. Python will also be a common entry point to high-performance capabilities. As we look toward the future, features in the C++ standard itself will become increasingly relevant for accessing the extreme parallelism of exascale platforms. This presentation will summarize the experiences of our team as we prepare for Aurora, exploring how to port applications to Aurora’s architecture and programming models, and distilling the challenges and best practices we’ve developed to date. oneAPI/SYCL and OpenMP are both critical models in these efforts, and while the ecosystem for Aurora has yet to mature, we’ve already had a great deal of success. Importantly, we are not passive recipients of programming models developed by others. Our team works not only with vendor-provided compilers and tools, but also develops improved open-source LLVM-based technologies that feed both open-source and vendor-provided capabilities. In addition, we actively participate in the standardization of OpenMP, SYCL, and C++. To conclude, I’ll share our thoughts on how these models can best develop in the future to support exascale-class systems." Watch the video: https://wp.me/p3RLHQ-lPT Learn more: https://www.iwocl.org/iwocl-2020/conference-program/ and https://www.anl.gov/topic/aurora Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Preparing to program Aurora at Exascale - Early experiences and future direct...

inside-BigData.com

In this deck, Greg Wahl from Advantech presents: Transforming Private 5G Networks. Advantech Networks & Communications Group is driving innovation in next-generation network solutions with their High Performance Servers. We provide business critical hardware to the world's leading telecom and networking equipment manufacturers with both standard and customized products. Our High Performance Servers are highly configurable platforms designed to balance the best in x86 server-class processing performance with maximum I/O and offload density. The systems are cost effective, highly available and optimized to meet next generation networking and media processing needs. “Advantech’s Networks and Communication Group has been both an innovator and trusted enabling partner in the telecommunications and network security markets for over a decade, designing and manufacturing products for OEMs that accelerate their network platform evolution and time to market.” Said Advantech Vice President of Networks & Communications Group, Ween Niu. “In the new IP Infrastructure era, we will be expanding our expertise in Software Defined Networking (SDN) and Network Function Virtualization (NFV), two of the essential conduits to 5G infrastructure agility making networks easier to install, secure, automate and manage in a cloud-based infrastructure.” In addition to innovation in air interface technologies and architecture extensions, 5G will also need a new generation of network computing platforms to run the emerging software defined infrastructure, one that provides greater topology flexibility, essential to deliver on the promises of high availability, high coverage, low latency and high bandwidth connections. This will open up new parallel industry opportunities through dedicated 5G network slices reserved for specific industries dedicated to video traffic, augmented reality, IoT, connected cars etc. 5G unlocks many new doors and one of the keys to its enablement lies in the elasticity and flexibility of the underlying infrastructure. Advantech’s corporate vision is to enable an intelligent planet. The company is a global leader in the fields of IoT intelligent systems and embedded platforms. To embrace the trends of IoT, big data, and artificial intelligence, Advantech promotes IoT hardware and software solutions with the Edge Intelligence WISE-PaaS core to assist business partners and clients in connecting their industrial chains. Advantech is also working with business partners to co-create business ecosystems that accelerate the goal of industrial intelligence." Watch the video: https://wp.me/p3RLHQ-lPQ * Company website: https://www.advantech.com/ * Solution page: https://www2.advantech.com/nc/newsletter/NCG/SKY/benefits.html Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Transforming Private 5G Networks

inside-BigData.com

In this deck from the Stanford HPC Conference, Katie Lewis from Lawrence Livermore National Laboratory presents: The Incorporation of Machine Learning into Scientific Simulations at Lawrence Livermore National Laboratory. "Scientific simulations have driven computing at Lawrence Livermore National Laboratory (LLNL) for decades. During that time, we have seen significant changes in hardware, tools, and algorithms. Today, data science, including machine learning, is one of the fastest growing areas of computing, and LLNL is investing in hardware, applications, and algorithms in this space. While the use of simulations to focus and understand experiments is well accepted in our community, machine learning brings new challenges that need to be addressed. I will explore applications for machine learning in scientific simulations that are showing promising results and further investigation that is needed to better understand its usefulness." Watch the video: https://youtu.be/NVwmvCWpZ6Y Learn more: https://computing.llnl.gov/research-area/machine-learning and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

The Incorporation of Machine Learning into Scientific Simulations at Lawrence...

inside-BigData.com

In this deck from the Stanford HPC Conference, DK Panda from Ohio State University presents: How to Achieve High-Performance, Scalable and Distributed DNN Training on Modern HPC Systems? "This talk will start with an overview of challenges being faced by the AI community to achieve high-performance, scalable and distributed DNN training on Modern HPC systems with both scale-up and scale-out strategies. After that, the talk will focus on a range of solutions being carried out in my group to address these challenges. The solutions will include: 1) MPI-driven Deep Learning, 2) Co-designing Deep Learning Stacks with High-Performance MPI, 3) Out-of- core DNN training, and 4) Hybrid (Data and Model) parallelism. Case studies to accelerate DNN training with popular frameworks like TensorFlow, PyTorch, MXNet and Caffe on modern HPC systems will be presented." Watch the video: https://youtu.be/LeUNoKZVuwQ Learn more: http://web.cse.ohio-state.edu/~panda.2/ and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...

inside-BigData.com

In this deck from the Stanford HPC Conference, Nick Nystrom and Paola Buitrago provide an update from the Pittsburgh Supercomputing Center. Nick Nystrom is Chief Scientist at the Pittsburgh Supercomputing Center (PSC). Nick is architect and PI for Bridges, PSC's flagship system that successfully pioneered the convergence of HPC, AI, and Big Data. He is also PI for the NIH Human Biomolecular Atlas Program’s HIVE Infrastructure Component and co-PI for projects that bring emerging AI technologies to research (Open Compass), apply machine learning to biomedical data for breast and lung cancer (Big Data for Better Health), and identify causal relationships in biomedical big data (the Center for Causal Discovery, an NIH Big Data to Knowledge Center of Excellence). His current research interests include hardware and software architecture, applications of machine learning to multimodal data (particularly for the life sciences) and to enhance simulation, and graph analytics. Watch the video: https://youtu.be/LWEU1L1o7yY Learn more: https://www.psc.edu/ and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...

inside-BigData.com

In this deck from the Stanford HPC Conference, Ryan Quick from Providentia Worldwide describes how DNNs can be used to improve EDA simulation runs. "Systems Intelligence relies on a variety of methods for providing insight into the core mechanisms for driving automated behavioral changes in self-healing command and control platforms. This talk reports on initial efforts with leveraging Semiconductor Electronic Design Automation (EDA) telemetry data from cross-domain sources including power, network, storage, nodes, and applications in neural networks as a driving method for insight into SI automation systems." Watch the video: https://youtu.be/2WbR8tq-XbM Learn more: http://www.providentiaworldwide.com/ and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

HPC Impact: EDA Telemetry Neural Networks

inside-BigData.com

In this deck from the Stanford HPC Conference, Nicole Xu from Stanford University describes how she transformed a common jellyfish into a bionic creature that is part animal and part machine. "Animal locomotion and bioinspiration have the potential to expand the performance capabilities of robots, but current implementations are limited. Mechanical soft robots leverage engineered materials and are highly controllable, but these biomimetic robots consume more power than corresponding animal counterparts. Biological soft robots from a bottom-up approach offer advantages such as speed and controllability but are limited to survival in cell media. Instead, biohybrid robots that comprise live animals and self- contained microelectronic systems leverage the animals’ own metabolism to reduce power constraints and body as an natural scaffold with damage tolerance. We demonstrate that by integrating onboard microelectronics into live jellyfish, we can enhance propulsion up to threefold, using only 10 mW of external power input to the microelectronics and at only a twofold increase in cost of transport to the animal. This robotic system uses 10 to 1000 times less external power per mass than existing swimming robots in literature and can be used in future applications for ocean monitoring to track environmental changes." Watch the video: https://youtu.be/HrmJFyvInj8 Learn more: https://sanfrancisco.cbslocal.com/2020/02/05/stanford-research-project-common-jellyfish-bionic-sea-creatures/ and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring

inside-BigData.com

In this deck from the Stanford HPC Conference, Peter Dueben from the European Centre for Medium-Range Weather Forecasts (ECMWF) presents: Machine Learning for Weather Forecasts. "I will present recent studies that use deep learning to learn the equations of motion of the atmosphere, to emulate model components of weather forecast models and to enhance usability of weather forecasts. I will than talk about the main challenges for the application of deep learning in cutting-edge weather forecasts and suggest approaches to improve usability in the future." Peter is contributing to the development and optimization of weather and climate models for modern supercomputers. He is focusing on a better understanding of model error and model uncertainty, on the use of reduced numerical precision that is optimised for a given level of model error, on global cloud- resolving simulations with ECMWF's forecast model, and the use of machine learning, and in particular deep learning, to improve the workflow and predictions. Peter has graduated in Physics and wrote his PhD thesis at the Max Planck Institute for Meteorology in Germany. He worked as Postdoc with Tim Palmer at the University of Oxford and has taken up a position as University Research Fellow of the Royal Society at the European Centre for Medium-Range Weather Forecasts (ECMWF) in 2017. Watch the video: https://youtu.be/ks3fkRj8Iqc Learn more: https://www.ecmwf.int/ and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Machine Learning for Weather Forecasts

inside-BigData.com

In this deck, Gilad Shainer from the HPC AI Advisory Council describes how this organization fosters innovation in the high performance computing community. "The HPC-AI Advisory Council’s mission is to bridge the gap between high-performance computing (HPC) and Artificial Intelligence (AI) use and its potential, bring the beneficial capabilities of HPC and AI to new users for better research, education, innovation and product manufacturing, bring users the expertise needed to operate HPC and AI systems, provide application designers with the tools needed to enable parallel computing, and to strengthen the qualification and integration of HPC and AI system products." Watch the video: https://wp.me/p3RLHQ-lNz Learn more: http://hpcadvisorycouncil.com Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

HPC AI Advisory Council Update

inside-BigData.com

Today RIKEN in Japan announced that the Fugaku supercomputer will be made available for research projects aimed to combat COVID-19. "Fugaku is currently being installed and is scheduled to be available to the public in 2021. However, faced with the devastating disaster unfolding before our eyes, RIKEN and MEXT decided to make a portion of the computational resources of Fugaku available for COVID-19-related projects ahead of schedule while continuing the installation process. Fugaku is being developed not only for the progress in science, but also to help build the society dubbed as the “Society 5.0” by the Japanese government, where all people will live safe and comfortable lives. The current initiative to fight against the novel coronavirus is driven by the philosophy behind the development of Fugaku." Initial Projects Exploring new drug candidates for COVID-19 by "Fugaku" Yasushi Okuno, RIKEN / Kyoto University Prediction of conformational dynamics of proteins on the surface of SARS-Cov-2 using Fugaku Yuji Sugita, RIKEN Simulation analysis of pandemic phenomena Nobuyasu Ito, RIKEN Fragment molecular orbital calculations for COVID-19 proteins Yuji Mochizuki, Rikkyo University

Fugaku Supercomputer joins fight against COVID-19

inside-BigData.com

In this deck from the Performance Optimisation and Productivity group, Lubomir Riha from IT4Innovations presents: Energy Efficient Computing using Dynamic Tuning. "We now live in a world of power-constrained architectures and systems and power consumption represents a significant cost factor in the overall HPC system economy. For these reasons, in recent years researchers, supercomputing centers and major vendors have developed new tools and methodologies to measure and optimize the energy consumption of large-scale high performance system installations. Due to the link between energy consumption, power consumption and execution time of an application executed by the final user, it is important for these tools and the methodology used to consider all these aspects, empowering the final user and the system administrator with the capability of finding the best configuration given different high level objectives. This webinar focused on tools designed to improve the energy-efficiency of HPC applications using a methodology of dynamic tuning of HPC applications, developed under the H2020 READEX project. The READEX methodology has been designed for exploiting the dynamic behaviour of software. At design time, different runtime situations (RTS) are detected and optimized system configurations are determined. RTSs with the same configuration are grouped into scenarios, forming the tuning model. At runtime, the tuning model is used to switch system configurations dynamically. The MERIC tool, that implements the READEX methodology, is presented. It supports manual or binary instrumentation of the analysed applications to simplify the analysis. This instrumentation is used to identify and annotate the significant regions in the HPC application. Automatic binary instrumentation annotates regions with significant runtime. Manual instrumentation, which can be combined with automatic, allows code developer to annotate regions of particular interest." Watch the video: https://wp.me/p3RLHQ-lJP Learn more: https://pop-coe.eu/blog/14th-pop-webinar-energy-efficient-computing-using-dynamic-tuning and https://code.it4i.cz/vys0053/meric Sign up for our insideHPC Newsletter: http://insidehpc.com/newslett

Energy Efficient Computing using Dynamic Tuning

inside-BigData.com

In this deck from GTC Digital, William Beaudin from DDN presents: HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD. Enabling high performance computing through the use of GPUs requires an incredible amount of IO to sustain application performance. We'll cover architectures that enable extremely scalable applications through the use of NVIDIA’s SuperPOD and DDN’s A3I systems. The NVIDIA DGX SuperPOD is a first-of-its-kind artificial intelligence (AI) supercomputing infrastructure. DDN A³I with the EXA5 parallel file system is a turnkey, AI data storage infrastructure for rapid deployment, featuring faster performance, effortless scale, and simplified operations through deeper integration. The combined solution delivers groundbreaking performance, deploys in weeks as a fully integrated system, and is designed to solve the world's most challenging AI problems. Watch the video: https://wp.me/p3RLHQ-lIV Learn more: https://www.ddn.com/download/nvidia-superpod-ddn-a3i-ai400-appliance-with-the-exa5-filesystem/ and https://www.nvidia.com/en-us/gtc/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD

inside-BigData.com

In this deck, Paul Isaacs from Linaro presents: State of ARM-based HPC. This talk provides an overview of applications and infrastructure services successfully ported to Aarch64 and benefiting from scale. "With its debut on the TOP500, the 125,000-core Astra supercomputer at New Mexico’s Sandia Labs uses Cavium ThunderX2 chips to mark Arm’s entry into the petascale world. In Japan, the Fujitsu A64FX Arm-based CPU in the pending Fugaku supercomputer has been optimized to achieve high-level, real-world application performance, anticipating up to one hundred times the application execution performance of the K computer. K was the first computer to top 10 petaflops in 2011." Watch the video: https://wp.me/p3RLHQ-lIT Learn more: https://www.linaro.org/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

State of ARM-based HPC

inside-BigData.com

Today Xilinx announced Versal Premium, the third series in the Versal ACAP portfolio. The Versal Premium series features highly integrated, networked and power-optimized cores and the industry’s highest bandwidth and compute density on an adaptable platform. Versal Premium is designed for the highest bandwidth networks operating in thermally and spatially constrained environments, as well as for cloud providers who need scalable, adaptable application acceleration. Versal is the industry’s first adaptive compute acceleration platform (ACAP), a revolutionary new category of heterogeneous compute devices with capabilities that far exceed those of conventional silicon architectures. Developed on TSMC’s 7-nanometer process technology, Versal Premium combines software programmability with dynamically configurable hardware acceleration and pre-engineered connectivity and security features to enable a faster time-to- market. The Versal Premium series delivers up to 3X higher throughput compared to current generation FPGAs, with built-in Ethernet, Interlaken, and cryptographic engines that enable fast and secure networks. The series doubles the compute density of currently deployed mainstream FPGAs and provides the adaptability to keep pace with increasingly diverse and evolving cloud and networking workloads. Learn more: https://insidehpc.com/2020/03/xilinx-announces-versal-premium-acap-for-network-and-cloud-acceleration/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Versal Premium ACAP for Network and Cloud Acceleration

inside-BigData.com

In this video from the Rice Oil & Gas Conference, Chin Fang from Zettar presents: Moving Massive Amounts of Data across Any Distance Efficiently. The objective of this talk is to present two on-going projects aiming at improving and ensuring highly efficient bulk transferring or streaming of massive amounts of data over digital connections across any distance. It examines the current state of the art, a few very common misconceptions, the differences among the three major type of data movement solutions, a current initiative attempting to improve the data movement efficiency from the ground up, and another multi-stage project that shows how to conduct long distance large scale data movement at speed and scale internationally. Both projects have real world motivations, e.g. the ambitious data transfer requirements of Linac Coherent Light Source II (LCLS-II) [1], a premier preparation project of the U.S. DOE Exascale Computing Initiative (ECI) [2]. Their immediate goals are described and explained, together with the solution used for each. Findings and early results are reported. Possible future works are outlined. Watch the video: https://wp.me/p3RLHQ-lBX Learn more: https://www.zettar.com/ and https://rice2020oghpc.rice.edu/program-2/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Zettar: Moving Massive Amounts of Data across Any Distance Efficiently

inside-BigData.com

In this deck from the Rice Oil & Gas Conference, Bradley McCredie from AMD presents: Scaling TCO in a Post Moore's Law Era. "While foundries bravely drive forward to overcome the technical and economic challenges posed by scaling to 5nm and beyond, Moore’s law alone can provide only a fraction of the performance / watt and performance / dollar gains needed to satisfy the demands of today’s high performance computing and artificial intelligence applications. To close the gap, multiple strategies are required. First, new levels of innovation and design efficiency will supplement technology gains to continue to deliver meaningful improvements in SoC performance. Second, heterogenous compute architectures will create x-factor increases of performance efficiency for the most critical applications. Finally, open software frameworks, APIs, and toolsets will enable broad ecosystems of application level innovation." Watch the video: Learn more: http://amd.com and https://rice2020oghpc.rice.edu/program-2/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Scaling TCO in a Post Moore's Era

inside-BigData.com

In this deck from the ECSS Symposium, Abe Stern from NVIDIA presents: CUDA-Python and RAPIDS for blazing fast scientific computing. "We will introduce Numba and RAPIDS for GPU programming in Python. Numba allows us to write just-in-time compiled CUDA code in Python, giving us easy access to the power of GPUs from a powerful high-level language. RAPIDS is a suite of tools with a Python interface for machine learning and dataframe operations. Together, Numba and RAPIDS represent a potent set of tools for rapid prototyping, development, and analysis for scientific computing. We will cover the basics of each library and go over simple examples to get users started. Finally, we will briefly highlight several other relevant libraries for GPU programming." Watch the video: https://wp.me/p3RLHQ-lvu Learn more: https://developer.nvidia.com/rapids and https://www.xsede.org/for-users/ecss/ecss-symposium Sign up for our insideHPC Newsletter: http://insidehp.com/newsletter

CUDA-Python and RAPIDS for blazing fast scientific computing

inside-BigData.com

In this deck from FOSDEM 2020, Colin Sauze from Aberystwyth University describes the development of a RaspberryPi cluster for teaching an introduction to HPC. "The motivation for this was to overcome four key problems faced by new HPC users: * The availability of a real HPC system and the effect running training courses can have on the real system, conversely the availability of spare resources on the real system can cause problems for the training course. * A fear of using a large and expensive HPC system for the first time and worries that doing something wrong might damage the system. * That HPC systems are very abstract systems sitting in data centres that users never see, it is difficult for them to understand exactly what it is they are using. * That new users fail to understand resource limitations, in part because of the vast resources in modern HPC systems a lot of mistakes can be made before running out of resources. A more resource constrained system makes it easier to understand this. The talk will also discuss some of the technical challenges in deploying an HPC environment to a Raspberry Pi and attempts to keep that environment as close to a "real" HPC as possible. The issue to trying to automate the installation process will also be covered." Learn more: https://github.com/colinsauze/pi_cluster and https://fosdem.org/2020/schedule/events/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Introducing HPC with a Raspberry Pi Cluster

inside-BigData.com

In this deck from ATPESC 2019, Ken Raffenetti from Argonne presents an overview of HPC interconnects. "The Argonne Training Program on Extreme-Scale Computing (ATPESC) provides intensive, two-week training on the key skills, approaches, and tools to design, implement, and execute computational science and engineering applications on current high-end computing systems and the leadership-class computing systems of the future." Watch the video: https://wp.me/p3RLHQ-luc Learn more: https://extremecomputingtraining.anl.gov/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Overview of HPC Interconnects

inside-BigData.com

More from inside-BigData.com (20)

Major Market Shifts in IT

Preparing to program Aurora at Exascale - Early experiences and future direct...

Transforming Private 5G Networks

The Incorporation of Machine Learning into Scientific Simulations at Lawrence...

How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...

Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...

HPC Impact: EDA Telemetry Neural Networks

Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring

Machine Learning for Weather Forecasts

HPC AI Advisory Council Update

Fugaku Supercomputer joins fight against COVID-19

Energy Efficient Computing using Dynamic Tuning

HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD

State of ARM-based HPC

Versal Premium ACAP for Network and Cloud Acceleration

Zettar: Moving Massive Amounts of Data across Any Distance Efficiently

Scaling TCO in a Post Moore's Era

CUDA-Python and RAPIDS for blazing fast scientific computing

Introducing HPC with a Raspberry Pi Cluster

Overview of HPC Interconnects

Recently uploaded

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Top 10 Most Downloaded Games on Play Store in 2024

SynarionITSolutions

A Principled Technologies deployment guide Conclusion Deploying VMware Cloud Foundation 5.1 on next gen Dell PowerEdge servers brings together critical virtualization capabilities and high-performing hardware infrastructure. Relying on our hands-on experience, this deployment guide offers a comprehensive roadmap that can guide your organization through the seamless integration of advanced VMware cloud solutions with the performance and reliability of Dell PowerEdge servers. In addition to the deployment efficiency, the Cloud Foundation 5.1 and PowerEdge solution delivered strong performance while running a MySQL database workload. By leveraging VMware Cloud Foundation 5.1 and PowerEdge servers, you could help your organization embrace cloud computing with confidence, potentially unlocking a new level of agility, scalability, and efficiency in your data center operations.

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Principled Technologies

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Increase engagement and revenue with Muvi Live Paywall! In this presentation, we will explore the five key benefits of using Muvi Live Paywall to monetize your live streams. You'll learn how Muvi Live Paywall can help you: Monetize your live content easily: Set up pay-per-view access to your live streams and start generating revenue from your content. Increase audience engagement: Provide exclusive, premium content behind the paywall to keep your viewers engaged. Gain valuable viewer insights: Track viewer data and analytics to better understand your audience and tailor your content accordingly. Reduce content piracy: Muvi Live Paywall's security features help protect your content from unauthorized distribution. Streamline your workflow: The all-in-one platform simplifies the process of managing and monetizing your live streams. With Muvi Live Paywall, you can take control of your live stream monetization and create a sustainable business model for your content. Learn more about Muvi Live Paywall and start generating revenue from your live streams today!

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Roshan Dwivedi

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Real Time Object Detection Using Open CV

Khem

Artificial Intelligence Chap.5 : Uncertainty

Khushali Kathiriya

MINDCTI Revenue Release Quarter One 2024

MIND CTI

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

With more memory available, system performance of three Dell devices increased, which can translate to a better user experience Conclusion When your system has plenty of RAM to meet your needs, you can efficiently access the applications and data you need to finish projects and to-do lists without sacrificing time and focus. Our test results show that with more memory available, three Dell PCs delivered better performance and took less time to complete the Procyon Office Productivity benchmark. These advantages translate to users being able to complete workflows more quickly and multitask more easily. Whether you need the mobility of the Latitude 5440, the creative capabilities of the Precision 3470, or the high performance of the OptiPlex Tower Plus 7010, configuring your system with more RAM can help keep processes running smoothly, enabling you to do more without compromising performance.

Boost PC performance: How more available memory can improve productivity

Principled Technologies

The presentation explores the development and application of artificial intelligence (AI) from its inception to its current status in the modern world. The term "artificial intelligence" was first coined by John McCarthy in 1956 to describe efforts to develop computer programs capable of performing tasks that typically require human intelligence. This concept was first introduced at a conference held at Dartmouth College, where programs demonstrated capabilities such as playing chess, proving theorems, and interpreting texts. In the early stages, Alan Turing contributed to the field by defining intelligence as the ability of a being to respond to certain questions intelligently, proposing what is now known as the Turing Test to evaluate the presence of intelligent behavior in machines. As the decades progressed, AI evolved significantly. The 1980s focused on machine learning, teaching computers to learn from data, leading to the development of models that could improve their performance based on their experiences. The 1990s and 2000s saw further advances in algorithms and computational power, which allowed for more sophisticated data analysis techniques, including data mining. By the 2010s, the proliferation of big data and the refinement of deep learning techniques enabled AI to become mainstream. Notable milestones included the success of Google's AlphaGo and advancements in autonomous vehicles by companies like Tesla and Waymo. A major theme of the presentation is the application of generative AI, which has been used for tasks such as natural language text generation, translation, and question answering. Generative AI uses large datasets to train models that can then produce new, coherent pieces of text or other media. The presentation also discusses the ethical implications and the need for regulation in AI, highlighting issues such as privacy, bias, and the potential for misuse. These concerns have prompted calls for comprehensive regulations to ensure the safe and equitable use of AI technologies. Artificial intelligence has also played a significant role in healthcare, particularly highlighted during the COVID-19 pandemic, where it was used in drug discovery, vaccine development, and analyzing the spread of the virus. The capabilities of AI in healthcare are vast, ranging from medical diagnostics to personalized medicine, demonstrating the technology's potential to revolutionize fields beyond just technical or consumer applications. In conclusion, AI continues to be a rapidly evolving field with significant implications for various aspects of society. The development from theoretical concepts to real-world applications illustrates both the potential benefits and the challenges that come with integrating advanced technologies into everyday life. The ongoing discussion about AI ethics and regulation underscores the importance of managing these technologies responsibly to maximize their their benefits while minimizing potential harms.

Artificial Intelligence: Facts and Myths

Joaquim Jorge

Manulife - Insurer Innovation Award 2024

The Digital Insurer

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Recently uploaded (20)

AWS Community Day CPH - Three problems of Terraform

Exploring the Future Potential of AI-Enabled Smartphone Processors

GenAI Risks & Security Meetup 01052024.pdf

Data Cloud, More than a CDP by Matt Robison

Top 10 Most Downloaded Games on Play Store in 2024

Deploy with confidence: VMware Cloud Foundation 5.1 on next gen Dell PowerEdg...

Apidays New York 2024 - The value of a flexible API Management solution for O...

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Real Time Object Detection Using Open CV

Artificial Intelligence Chap.5 : Uncertainty

MINDCTI Revenue Release Quarter One 2024

How to Troubleshoot Apps for the Modern Connected Worker

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Boost PC performance: How more available memory can improve productivity

Artificial Intelligence: Facts and Myths

Manulife - Insurer Innovation Award 2024

Automating Google Workspace (GWS) & more with Apps Script

Optimizing Lustre and GPFS with DDN

1. Optimizing Lustre and GPFS Solutions with DDN Robert Triendl VP of Worldwide HPC Strategy, DataDirect Networks

2. 2! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com File Systems @ DDN

3. 3! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com File System Basics •  File system are where your data lives •  File systems are complex software level technologies… •  … so there are always surprises! •  There are huge differences in performance, functionality, and reliability •  When it comes to performance, no file system fits all requirements

4. 4! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com Test and Benchmark Labs

5. 5! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com GTLS | Benchmark Lab Sites EMEA Lab Dusseldorf, Germany Asia Pacific Lab Tokyo, Japan East Coast Lab Columbia, MD West Coast Lab Sunnyvale, CA

6. 6! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com DDN and Lustre •  Started with Lustre 0.6, and the first commercial Lustre support contract with CFS! •  Over 250 EXAScaler customers worldwide today and many more using DDN storage for Lustre •  Customers in many industries (HPC centers, Large Experimental Facilities, Oil & Gas, Life Science, Automotive, etc.) •  Very broad set of applications supported

7. 7! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com Corp Data 4% Government Security 17% Research Data Analysis, 28%HPC Archive 18% HPC Work 20% HPC Work Corp 12% Project Quota Metadata Perf SSD Acceleration Fine-Grained Monitoring NFS/CIFS Access Management Connectors Object/Cloud Links Data Management Backup/Replication HSM Client Performance Cluster Integration Large I/O IME Caching Security Features Lustre WAN RAS Small File I/O

8. 8! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com DDN Open Source Lustre Contributions 0 20 40 60 80 100 120 140 160 180 2.1 2.4.0 2.3.50-2.4.0 2.5.0 2.5.50-2.6.0 EMC CEA SUSE Bull Other Cray LLNN Xyratex DDN

9. 9! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com Large RPC Size Effects 0%! 20%! 40%! 60%! 80%! 100%! 120%! 0! 100! 200! 300! 400! 500! 600! 700! Number of Process! WRITE! 7.2KSAS(1MB RPC)! 7.2KSAS(4MB RPC)! SSD(1MB RPC)! 0%! 20%! 40%! 60%! 80%! 100%! 120%! 0! 100! 200! 300! 400! 500! 600! 700! Number of Process! READ! 7.2KSAS(1MB RPC)! 7.2KSAS(4MB RPC)! SSD(1MB RPC)!

10. 10! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com •  Limited single client scaling •  Good scaling with clock speed •  Good Scaling with core count and HT •  Great Scaling with DNE •  Limita<ons on Dir Creates (TBD) Lustre Metadata

11. 11! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com mmap() I/O Performance Improvements 0! 100! 200! 300! 400! 500! lustre-1.8.9! lustre-2.5.2! DDN branch! mmap() Read Performance ! (1MB block size)! 0! 100! 200! 300! 400! 500! 32K! 128K! 512K! 1024K! mmap() Read Performance! Lustre-1.8.9! DDN branch!

12. 12! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com EXAScaler Monitoring OSS/MDS! collectd! Lustre client! DDN monitoring plugin! graphite! Monitoring Server! collectd! Graphite plugin! UDP(TCP)/IP based small text message transfer graphite! •  Lightweight •  Near real-‐<me •  Massive scale •  Customizable •  File system, OST Pool, OST/MDT stats, etc. •  JOB ID, UID/GID, aggregation of application's stats, etc. •  Archive of data by policy

13. 13! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com EXAScaler Monitoring •  Running in TITECH –  over 112 Object Storage Targets across –  1700 clients •  That’s around 1M statistics •  Need to store every few seconds •  Demo of over 10M stats at DDN Booth

14. 14! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com VMs on GRIDScaler 256 VMs on 16 Clients 0! 2000! 4000! 6000! 8000! 10000! 12000! 14000! 16000! 1! 2! 4! 8! 16! 32! 64! 128! 256! Throughput(MB/sec)! Number of Process! 0! 2000! 4000! 6000! 8000! 10000! 12000! 1! 2! 4! 8! 16! 32! 64! 128! 256! Throughput(MB/sec)! Number of Process!

15. 15! © 2014 DataDirect Networks, Inc. * Other names and brands may be claimed as the property of others. Any statements or representations around future events are subject to change. ddn.com 0! 100! 200! 300! 400! 500! 600! 700! 800! 900! 1000! 1! 10! 20! 30! 40! Total Bandwidth! Read Bandwidth! Write Bandwidth! GRIDScaler for OpenStack vbench Results 0! 1000! 2000! 3000! 4000! 5000! 6000! 1! 10! 20! 30! 40! Total IOPS! Read IOPS! Write IOPS!

Optimizing Lustre and GPFS with DDN

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Optimizing Lustre and GPFS with DDN

Similar to Optimizing Lustre and GPFS with DDN (20)

More from inside-BigData.com

More from inside-BigData.com (20)

Recently uploaded

Recently uploaded (20)

Optimizing Lustre and GPFS with DDN