Creating a Better Infrastructure to Manage Big Data

•

1 like•650 views

"While traditional HPC is at the core of supercomputing, recent advancements are allowing HPC centers to offer new and exciting compute services to their users and other customers. Adaptive Computing will discuss two such advancements, topology-based scheduling and HPC cloud, and how together they provide an infrastructure that allow HPC centers to provide Big Data and services along with traditional offerings." Over the course of this talk, Trev Harmon from Adaptive Computing looks back to the utility computing vision of Douglas Parkhill and proposes an application-centric workflow for the future that fulfills that vision across many disciplines of computing. Watch the presentation video: http://wp.me/p3RLHQ-aX3

Technology Education

Creating a Better Infrastructure
to Manage Big Data
SC’13 Exhibitor Forum
Software for HPC III
Trev Harmon
Principal Solution Strategist
Adaptive Computing

1

© 2013 ADAPTIVE COMPUTING, INC.

21 Nov 2013
4:00-4:30pm

Let’s Turn Back the Clock

Denver,
Colorado

5

2010

2013

© 2013 ADAPTIVE COMPUTING, INC.

Douglas Parkhill’s Vision

6

© 2013 ADAPTIVE COMPUTING, INC.

An Over-simplification

Visualization

HPC

Technical
Computing

Computing
Big Data
Cloud

9

© 2013 ADAPTIVE COMPUTING, INC.

Scientific
Computing

Multiple Fields
Multiple Users

10

© 2013 ADAPTIVE COMPUTING, INC.

Computing Workflows

Visualization

HPC

11

© 2013 ADAPTIVE COMPUTING, INC.

Computing Workflows

12

Scientific
Computing

Big Data

Visualization

Cloud

HPC

© 2013 ADAPTIVE COMPUTING, INC.

Really, we’re optimizing…

Job
Needs
Available
Resources
13

© 2013 ADAPTIVE COMPUTING, INC.

Topology and Company

Topology-Awareness
Data-Locality Awareness
Hardware Preferences
14

© 2013 ADAPTIVE COMPUTING, INC.

Another Merger

Cloud

Scientific
Computing

Big Data

HPC
Cloud
Visualization

15

© 2013 ADAPTIVE COMPUTING, INC.

HPC

Defining HPC Cloud

ds…
en
lly.
rea
dep
It
m,
the
l of
ll al
we

16

Cloud-based HPC ?
Cloud-bursting ?
VM-encapsulated Jobs ?
© 2013 ADAPTIVE COMPUTING, INC.

The Future of Computing
Is Founded on HPC
Principles and Concepts.

17

© 2013 ADAPTIVE COMPUTING, INC.

Douglas Parkhill’s Vision

18

© 2013 ADAPTIVE COMPUTING, INC.

Attribution & Citation
▪ 
▪ 
▪ 

21

Denver Photo:
http://www.flickr.com/photos/dagpeak/
Douglas Parkhill Photo: http://www.spectralumni.ca/CommExpress_pdf/
cemay84e.pdf
Topology Map:
US Geological Survey, US Department of the Interior/USGS

© 2013 ADAPTIVE COMPUTING, INC.

Elastic DR: a solution architecture that aims to optimally balance cost and recovery time via three core principles that are germane the cloud world: On-Demand: The disaster recovery cloud can be provisioned on any availability zone, region, or public/private cloud through Cloudify's cloud-agnostic bootstrapping mechanism. Elastic: The ability to automatically provision resources in the recovery cloud in case of disaster while eliminating the need for idle resources in normal scenarios, thereby fully profiting from the pay-per-use pricing model of clouds. Flexible RTO/RPO: The architecture can be easily extended from a warm DR to a hot DR pattern through enabling/disabling application recipes. This allows us to exploit economies of scale that the cloud provides by matching the number of recipes/tiers to provision (in the recovery cloud) against the recovery time/point objective for our disaster recovery strategy

Data Orchestration for AI, Big Data, and Cloud

Alluxio, Inc.

Getting The ROI Out

Intel Corporation

Hadoop performance modeling for job estimation and resource provisioning Do Your Projects With Technology Experts To Get this projects Call : 9566355386 / 99625 88976 Web : http://www.lemenizinfotech.com Web : http://www.ieeemaster.com Mail : projects@lemenizinfotech.com Blog : http://ieeeprojectspondicherry.weebly.com Blog : http://www.ieeeprojectsinpondicherry.blogspot.in/ Youtube:https://www.youtube.com/watch?v=eesBNUnKvws

IRJET- A Review on K-Means++ Clustering Algorithm and Cloud Computing wit...

IRJET Journal

2018 19 Cloudcomputing

Rajesh Math

Why Infrastructure matters?!

Gabi Bauer

The emergence of social, mobile, cloud, big data and analytics are fundamentally changing how we live, work and interact. Mobile devices are ubiquitous. Changing consumer behaviors, supplanting PCs, generating massive amounts of data and putting new demands on the enterprise to not only support these devices but to adjust the way they do business. Social technologies are changing the way we interact, communicate and share information – equally generating vast amounts of data and impacting business as they try to unlock the full potential social has to offer. Cloud technologies bring new scale and efficiency to service delivery and enable more agile ways of doing business and drive business model innovation. For companies, It also brings information and applications to people at the right time and place. All of these trends are fueling an explosion of data. Not only do enterprises need to store, manage and secure this data, they also need to derive meaningful insight from these vast amounts of data. Data is the basis of significant opportunity and a source of competitive advantage for all organizations. Data is a new economic asset, the next natural resource. These trends are spawning new workloads, business processes and technology deployments that are putting unprecedented demands on our IT environments.

Your Self-Driving Car - How Did it Get So Smart?

Hortonworks

As we all can appreciate, “teaching” a vehicle to drive under the full range of conditions it will encounter (i.e. road conditions, weather conditions, behavior of other vehicles) is a daunting proposition. If merely the thought of this makes you nervous, you’re not alone – according to the American Automobile Association (AAA), 75 percent of consumers are not yet ready to embrace self-driving cars. However, that is the very challenge facing automakers – teaching vehicles to unfailingly assess and respond to any combination of operational conditions “on-the-fly” through discrete rules (algorithms) governing a vehicle’s behavior. Join Hortonworks and NorCom at the upcoming webinar as we discuss: •Evolution of the autonomous driving •Traditional data management approaches and main challenges associated with them •How NorCom and Hortonworks can address those challenges and accelerate the pace of autonomous development

B04 06 0918International Journal of Engineering Inventions www.ijeijournal.com

Cloud computing

NEHA GAUTAM

Assignment on Cloud Computing

Al Shahriar

Where the Warehouse Ends: A New Age of Information Access

Inside Analysis

The Briefing Room with Barry Devlin and Composite Software Live Webcast May 21, 2013 All good things must come to an end, and even though the data warehouse will remain a prominent force in the information age, the handwriting is all over the enterprise: the center of gravity is moving. Whether due to Big Data or real-time demands, Cloud computing or globalization, today's leading organizations have analytical needs that the warehouse simply cannot accommodate. That's why data virtualization continues to attract attention. Register for this episode of The Briefing Room to hear veteran Analyst Barry Devlin explain why the traditional model for data warehousing is being outmoded by a range of more flexible methods for accessing and analyzing information assets. He'll be briefed by David Besemer of Composite Software who will discuss how his company's data virtualization platform can be used to provide access to all manner of information sources, including data warehouses, Big Data silos, as well as partner and public data sources on demand. Visit: http://www.insideanalysis.com

Hadoop 2.0 - Solving the Data Quality Challenge

Inside Analysis

The Briefing Room with Dr. Claudia Imhoff and RedPoint Global Live Webcast on July 22, 2014 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=7bb4cbc33402c3b5f649343052cb9a6d Whether data is big or small, quality remains the critical characteristic. While traditional approaches to cleansing data have made strides, nonetheless, data quality remains a serious hurdle for all organizations. This is especially true for identity resolution in customer data, but also for a range of other data sets, including social, supply chain, financial and other domains. One of the most promising approaches for solving this decades-old challenge incorporates the power of massive parallel processing, a la Hadoop. Register for this episode of The Briefing Room to learn from veteran Analyst Dr. Claudia Imhoff, who will explain how Hadoop 2.0 and its YARN architecture can make a serious impact on the previously intractable problem of data quality. She’ll be briefed by George Corugedo of RedPoint Global, who will show how his company’s platform can serve as a super-charged marshaling area for accessing, cleansing and delivering high-quality data. He’ll explain how RedPoint was one of the first applications to be certified for running on YARN, which is the latest rendition of the now-ubiquitous Hadoop. Visit InsideAnlaysis.com for more information.

Big Workflow Launch from Adaptive Computing

inside-BigData.com

In this slidecast, Rob Clyde from Adaptive Computing describes Big Workflow-- the convergence of Cloud, Big Data, and HPC in enterprise computing. "The explosion of big data, coupled with the collisions of HPC and cloud, is driving the evolution of big data analytics," said Rob Clyde, CEO of Adaptive Computing. "A Big Workflow approach to big data not only delivers business intelligence more rapidly, accurately and cost effectively, but also provides a distinct competitive advantage. We are confident that Big Workflow will enable enterprises across all industries to leverage big data that inspires game-changing, data-driven decisions." Learn more: http://bit.ly/bigworkflow Watch the video presentation: http://wp.me/p3RLHQ-bp2

Top 10 ways BigInsights BigIntegrate and BigQuality will improve your life

IBM Analytics

Cloud computing ebook

Ankit Gupta

What's hot

Introducing the 2013 Geospatial Portfolio_ Alastair Ritchie - Intergraph Geos...

IMGS

More on my LinkedIn SummaryMike Eghtebas

Realworld Powersavings for the Cloud Customer

cloudcampghent

BK2015 åpningen Esri uk - dominic stubbins - the geographic renaissance

Geodata AS

GPUdb: A Distributed Database for Many-Core Devices

inside-BigData.com

Twister4Azure - Iterative MapReduce for Azure Cloud

Thilina Gunarathne

Meetup at AI NextCon 2019: In-Stream data process, Data Orchestration & More

Alluxio, Inc.

Greenplum for Kubernetes PGConf india 2019

Goutam Tadi

What's hot (8)

Introducing the 2013 Geospatial Portfolio_ Alastair Ritchie - Intergraph Geos...

Similar to Creating a Better Infrastructure to Manage Big Data

IRJET- Comparatively Analysis on K-Means++ and Mini Batch K-Means Clustering ...

IRJET Journal

Hadoop performance modeling for job estimation and resource provisioning

LeMeniz Infotech

IRJET- A Review on K-Means++ Clustering Algorithm and Cloud Computing wit...

IRJET Journal

2018 19 Cloudcomputing

Rajesh Math

Why Infrastructure matters?!

Gabi Bauer

Your Self-Driving Car - How Did it Get So Smart?

Hortonworks

B04 06 0918International Journal of Engineering Inventions www.ijeijournal.com

Cloud computing

NEHA GAUTAM

Assignment on Cloud Computing

Al Shahriar

Where the Warehouse Ends: A New Age of Information Access

Inside Analysis

Hadoop 2.0 - Solving the Data Quality Challenge

Inside Analysis

Big Workflow Launch from Adaptive Computing

inside-BigData.com

Top 10 ways BigInsights BigIntegrate and BigQuality will improve your life

IBM Analytics

Cloud computing ebook

Ankit Gupta

The TCO Calculator - Estimate the True Cost of Hadoop

MapR Technologies

Google Cloud Data Platform - Why Google for Data Analysis?

Andreas Raible

Cloud expo june 2013: Building a Real Time Analytics Platform on Big Data in ...

Sanjay Sharma

Big data with hadoop

Anusha sweety

What's New in Moab 8.0

inside-BigData.com

Cloud Computing TechnologyAhmed Al Salih

Similar to Creating a Better Infrastructure to Manage Big Data (20)

IRJET- Comparatively Analysis on K-Means++ and Mini Batch K-Means Clustering ...

Hadoop performance modeling for job estimation and resource provisioning

IRJET- A Review on K-Means++ Clustering Algorithm and Cloud Computing wit...

2018 19 Cloudcomputing

Why Infrastructure matters?!

Your Self-Driving Car - How Did it Get So Smart?

B04 06 0918

Cloud computing

Assignment on Cloud Computing

Where the Warehouse Ends: A New Age of Information Access

Hadoop 2.0 - Solving the Data Quality Challenge

Big Workflow Launch from Adaptive Computing

Top 10 ways BigInsights BigIntegrate and BigQuality will improve your life

Cloud computing ebook

The TCO Calculator - Estimate the True Cost of Hadoop

Google Cloud Data Platform - Why Google for Data Analysis?

Cloud expo june 2013: Building a Real Time Analytics Platform on Big Data in ...

Big data with hadoop

What's New in Moab 8.0

Cloud Computing Technology

More from inside-BigData.com

Major Market Shifts in IT

inside-BigData.com

In this deck from the Stanford HPC Conference, Shahin Khan from OrionX describes major market Shifts in IT. "We will discuss the digital infrastructure of the future enterprise and the state of these trends." "We work with clients on the impact of Digital Transformation (DX) on them, their customers, and their messages. Generally, they want to track, in one place, trends like IoT, 5G, AI, Blockchain, and Quantum Computing. And they want to know what these trends mean, how they affect each other, and when they demand action, and how to formulate and execute an effective plan. If that describes you, we can help." Watch the video: https://wp.me/p3RLHQ-lPP Learn more: http://orionx.net and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Preparing to program Aurora at Exascale - Early experiences and future direct...

inside-BigData.com

In this deck from IWOCL / SYCLcon 2020, Hal Finkel from Argonne National Laboratory presents: Preparing to program Aurora at Exascale - Early experiences and future directions. "Argonne National Laboratory’s Leadership Computing Facility will be home to Aurora, our first exascale supercomputer. Aurora promises to take scientific computing to a whole new level, and scientists and engineers from many different fields will take advantage of Aurora’s unprecedented computational capabilities to push the boundaries of human knowledge. In addition, Aurora’s support for advanced machine-learning and big-data computations will enable scientific workflows incorporating these techniques along with traditional HPC algorithms. Programming the state-of-the-art hardware in Aurora will be accomplished using state-of-the-art programming models. Some of these models, such as OpenMP, are long-established in the HPC ecosystem. Other models, such as Intel’s oneAPI, based on SYCL, are relatively-new models constructed with the benefit of significant experience. Many applications will not use these models directly, but rather, will use C++ abstraction libraries such as Kokkos or RAJA. Python will also be a common entry point to high-performance capabilities. As we look toward the future, features in the C++ standard itself will become increasingly relevant for accessing the extreme parallelism of exascale platforms. This presentation will summarize the experiences of our team as we prepare for Aurora, exploring how to port applications to Aurora’s architecture and programming models, and distilling the challenges and best practices we’ve developed to date. oneAPI/SYCL and OpenMP are both critical models in these efforts, and while the ecosystem for Aurora has yet to mature, we’ve already had a great deal of success. Importantly, we are not passive recipients of programming models developed by others. Our team works not only with vendor-provided compilers and tools, but also develops improved open-source LLVM-based technologies that feed both open-source and vendor-provided capabilities. In addition, we actively participate in the standardization of OpenMP, SYCL, and C++. To conclude, I’ll share our thoughts on how these models can best develop in the future to support exascale-class systems." Watch the video: https://wp.me/p3RLHQ-lPT Learn more: https://www.iwocl.org/iwocl-2020/conference-program/ and https://www.anl.gov/topic/aurora Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Transforming Private 5G Networks

inside-BigData.com

In this deck, Greg Wahl from Advantech presents: Transforming Private 5G Networks. Advantech Networks & Communications Group is driving innovation in next-generation network solutions with their High Performance Servers. We provide business critical hardware to the world's leading telecom and networking equipment manufacturers with both standard and customized products. Our High Performance Servers are highly configurable platforms designed to balance the best in x86 server-class processing performance with maximum I/O and offload density. The systems are cost effective, highly available and optimized to meet next generation networking and media processing needs. “Advantech’s Networks and Communication Group has been both an innovator and trusted enabling partner in the telecommunications and network security markets for over a decade, designing and manufacturing products for OEMs that accelerate their network platform evolution and time to market.” Said Advantech Vice President of Networks & Communications Group, Ween Niu. “In the new IP Infrastructure era, we will be expanding our expertise in Software Defined Networking (SDN) and Network Function Virtualization (NFV), two of the essential conduits to 5G infrastructure agility making networks easier to install, secure, automate and manage in a cloud-based infrastructure.” In addition to innovation in air interface technologies and architecture extensions, 5G will also need a new generation of network computing platforms to run the emerging software defined infrastructure, one that provides greater topology flexibility, essential to deliver on the promises of high availability, high coverage, low latency and high bandwidth connections. This will open up new parallel industry opportunities through dedicated 5G network slices reserved for specific industries dedicated to video traffic, augmented reality, IoT, connected cars etc. 5G unlocks many new doors and one of the keys to its enablement lies in the elasticity and flexibility of the underlying infrastructure. Advantech’s corporate vision is to enable an intelligent planet. The company is a global leader in the fields of IoT intelligent systems and embedded platforms. To embrace the trends of IoT, big data, and artificial intelligence, Advantech promotes IoT hardware and software solutions with the Edge Intelligence WISE-PaaS core to assist business partners and clients in connecting their industrial chains. Advantech is also working with business partners to co-create business ecosystems that accelerate the goal of industrial intelligence." Watch the video: https://wp.me/p3RLHQ-lPQ * Company website: https://www.advantech.com/ * Solution page: https://www2.advantech.com/nc/newsletter/NCG/SKY/benefits.html Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

The Incorporation of Machine Learning into Scientific Simulations at Lawrence...

inside-BigData.com

In this deck from the Stanford HPC Conference, Katie Lewis from Lawrence Livermore National Laboratory presents: The Incorporation of Machine Learning into Scientific Simulations at Lawrence Livermore National Laboratory. "Scientific simulations have driven computing at Lawrence Livermore National Laboratory (LLNL) for decades. During that time, we have seen significant changes in hardware, tools, and algorithms. Today, data science, including machine learning, is one of the fastest growing areas of computing, and LLNL is investing in hardware, applications, and algorithms in this space. While the use of simulations to focus and understand experiments is well accepted in our community, machine learning brings new challenges that need to be addressed. I will explore applications for machine learning in scientific simulations that are showing promising results and further investigation that is needed to better understand its usefulness." Watch the video: https://youtu.be/NVwmvCWpZ6Y Learn more: https://computing.llnl.gov/research-area/machine-learning and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...

inside-BigData.com

In this deck from the Stanford HPC Conference, DK Panda from Ohio State University presents: How to Achieve High-Performance, Scalable and Distributed DNN Training on Modern HPC Systems? "This talk will start with an overview of challenges being faced by the AI community to achieve high-performance, scalable and distributed DNN training on Modern HPC systems with both scale-up and scale-out strategies. After that, the talk will focus on a range of solutions being carried out in my group to address these challenges. The solutions will include: 1) MPI-driven Deep Learning, 2) Co-designing Deep Learning Stacks with High-Performance MPI, 3) Out-of- core DNN training, and 4) Hybrid (Data and Model) parallelism. Case studies to accelerate DNN training with popular frameworks like TensorFlow, PyTorch, MXNet and Caffe on modern HPC systems will be presented." Watch the video: https://youtu.be/LeUNoKZVuwQ Learn more: http://web.cse.ohio-state.edu/~panda.2/ and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...

inside-BigData.com

In this deck from the Stanford HPC Conference, Nick Nystrom and Paola Buitrago provide an update from the Pittsburgh Supercomputing Center. Nick Nystrom is Chief Scientist at the Pittsburgh Supercomputing Center (PSC). Nick is architect and PI for Bridges, PSC's flagship system that successfully pioneered the convergence of HPC, AI, and Big Data. He is also PI for the NIH Human Biomolecular Atlas Program’s HIVE Infrastructure Component and co-PI for projects that bring emerging AI technologies to research (Open Compass), apply machine learning to biomedical data for breast and lung cancer (Big Data for Better Health), and identify causal relationships in biomedical big data (the Center for Causal Discovery, an NIH Big Data to Knowledge Center of Excellence). His current research interests include hardware and software architecture, applications of machine learning to multimodal data (particularly for the life sciences) and to enhance simulation, and graph analytics. Watch the video: https://youtu.be/LWEU1L1o7yY Learn more: https://www.psc.edu/ and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

HPC Impact: EDA Telemetry Neural Networks

inside-BigData.com

In this deck from the Stanford HPC Conference, Ryan Quick from Providentia Worldwide describes how DNNs can be used to improve EDA simulation runs. "Systems Intelligence relies on a variety of methods for providing insight into the core mechanisms for driving automated behavioral changes in self-healing command and control platforms. This talk reports on initial efforts with leveraging Semiconductor Electronic Design Automation (EDA) telemetry data from cross-domain sources including power, network, storage, nodes, and applications in neural networks as a driving method for insight into SI automation systems." Watch the video: https://youtu.be/2WbR8tq-XbM Learn more: http://www.providentiaworldwide.com/ and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring

inside-BigData.com

In this deck from the Stanford HPC Conference, Nicole Xu from Stanford University describes how she transformed a common jellyfish into a bionic creature that is part animal and part machine. "Animal locomotion and bioinspiration have the potential to expand the performance capabilities of robots, but current implementations are limited. Mechanical soft robots leverage engineered materials and are highly controllable, but these biomimetic robots consume more power than corresponding animal counterparts. Biological soft robots from a bottom-up approach offer advantages such as speed and controllability but are limited to survival in cell media. Instead, biohybrid robots that comprise live animals and self- contained microelectronic systems leverage the animals’ own metabolism to reduce power constraints and body as an natural scaffold with damage tolerance. We demonstrate that by integrating onboard microelectronics into live jellyfish, we can enhance propulsion up to threefold, using only 10 mW of external power input to the microelectronics and at only a twofold increase in cost of transport to the animal. This robotic system uses 10 to 1000 times less external power per mass than existing swimming robots in literature and can be used in future applications for ocean monitoring to track environmental changes." Watch the video: https://youtu.be/HrmJFyvInj8 Learn more: https://sanfrancisco.cbslocal.com/2020/02/05/stanford-research-project-common-jellyfish-bionic-sea-creatures/ and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Machine Learning for Weather Forecasts

inside-BigData.com

In this deck from the Stanford HPC Conference, Peter Dueben from the European Centre for Medium-Range Weather Forecasts (ECMWF) presents: Machine Learning for Weather Forecasts. "I will present recent studies that use deep learning to learn the equations of motion of the atmosphere, to emulate model components of weather forecast models and to enhance usability of weather forecasts. I will than talk about the main challenges for the application of deep learning in cutting-edge weather forecasts and suggest approaches to improve usability in the future." Peter is contributing to the development and optimization of weather and climate models for modern supercomputers. He is focusing on a better understanding of model error and model uncertainty, on the use of reduced numerical precision that is optimised for a given level of model error, on global cloud- resolving simulations with ECMWF's forecast model, and the use of machine learning, and in particular deep learning, to improve the workflow and predictions. Peter has graduated in Physics and wrote his PhD thesis at the Max Planck Institute for Meteorology in Germany. He worked as Postdoc with Tim Palmer at the University of Oxford and has taken up a position as University Research Fellow of the Royal Society at the European Centre for Medium-Range Weather Forecasts (ECMWF) in 2017. Watch the video: https://youtu.be/ks3fkRj8Iqc Learn more: https://www.ecmwf.int/ and http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

HPC AI Advisory Council Update

inside-BigData.com

In this deck, Gilad Shainer from the HPC AI Advisory Council describes how this organization fosters innovation in the high performance computing community. "The HPC-AI Advisory Council’s mission is to bridge the gap between high-performance computing (HPC) and Artificial Intelligence (AI) use and its potential, bring the beneficial capabilities of HPC and AI to new users for better research, education, innovation and product manufacturing, bring users the expertise needed to operate HPC and AI systems, provide application designers with the tools needed to enable parallel computing, and to strengthen the qualification and integration of HPC and AI system products." Watch the video: https://wp.me/p3RLHQ-lNz Learn more: http://hpcadvisorycouncil.com Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Fugaku Supercomputer joins fight against COVID-19

inside-BigData.com

Today RIKEN in Japan announced that the Fugaku supercomputer will be made available for research projects aimed to combat COVID-19. "Fugaku is currently being installed and is scheduled to be available to the public in 2021. However, faced with the devastating disaster unfolding before our eyes, RIKEN and MEXT decided to make a portion of the computational resources of Fugaku available for COVID-19-related projects ahead of schedule while continuing the installation process. Fugaku is being developed not only for the progress in science, but also to help build the society dubbed as the “Society 5.0” by the Japanese government, where all people will live safe and comfortable lives. The current initiative to fight against the novel coronavirus is driven by the philosophy behind the development of Fugaku." Initial Projects Exploring new drug candidates for COVID-19 by "Fugaku" Yasushi Okuno, RIKEN / Kyoto University Prediction of conformational dynamics of proteins on the surface of SARS-Cov-2 using Fugaku Yuji Sugita, RIKEN Simulation analysis of pandemic phenomena Nobuyasu Ito, RIKEN Fragment molecular orbital calculations for COVID-19 proteins Yuji Mochizuki, Rikkyo University

Energy Efficient Computing using Dynamic Tuning

inside-BigData.com

In this deck from the Performance Optimisation and Productivity group, Lubomir Riha from IT4Innovations presents: Energy Efficient Computing using Dynamic Tuning. "We now live in a world of power-constrained architectures and systems and power consumption represents a significant cost factor in the overall HPC system economy. For these reasons, in recent years researchers, supercomputing centers and major vendors have developed new tools and methodologies to measure and optimize the energy consumption of large-scale high performance system installations. Due to the link between energy consumption, power consumption and execution time of an application executed by the final user, it is important for these tools and the methodology used to consider all these aspects, empowering the final user and the system administrator with the capability of finding the best configuration given different high level objectives. This webinar focused on tools designed to improve the energy-efficiency of HPC applications using a methodology of dynamic tuning of HPC applications, developed under the H2020 READEX project. The READEX methodology has been designed for exploiting the dynamic behaviour of software. At design time, different runtime situations (RTS) are detected and optimized system configurations are determined. RTSs with the same configuration are grouped into scenarios, forming the tuning model. At runtime, the tuning model is used to switch system configurations dynamically. The MERIC tool, that implements the READEX methodology, is presented. It supports manual or binary instrumentation of the analysed applications to simplify the analysis. This instrumentation is used to identify and annotate the significant regions in the HPC application. Automatic binary instrumentation annotates regions with significant runtime. Manual instrumentation, which can be combined with automatic, allows code developer to annotate regions of particular interest." Watch the video: https://wp.me/p3RLHQ-lJP Learn more: https://pop-coe.eu/blog/14th-pop-webinar-energy-efficient-computing-using-dynamic-tuning and https://code.it4i.cz/vys0053/meric Sign up for our insideHPC Newsletter: http://insidehpc.com/newslett

HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD

inside-BigData.com

In this deck from GTC Digital, William Beaudin from DDN presents: HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD. Enabling high performance computing through the use of GPUs requires an incredible amount of IO to sustain application performance. We'll cover architectures that enable extremely scalable applications through the use of NVIDIA’s SuperPOD and DDN’s A3I systems. The NVIDIA DGX SuperPOD is a first-of-its-kind artificial intelligence (AI) supercomputing infrastructure. DDN A³I with the EXA5 parallel file system is a turnkey, AI data storage infrastructure for rapid deployment, featuring faster performance, effortless scale, and simplified operations through deeper integration. The combined solution delivers groundbreaking performance, deploys in weeks as a fully integrated system, and is designed to solve the world's most challenging AI problems. Watch the video: https://wp.me/p3RLHQ-lIV Learn more: https://www.ddn.com/download/nvidia-superpod-ddn-a3i-ai400-appliance-with-the-exa5-filesystem/ and https://www.nvidia.com/en-us/gtc/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

State of ARM-based HPC

inside-BigData.com

In this deck, Paul Isaacs from Linaro presents: State of ARM-based HPC. This talk provides an overview of applications and infrastructure services successfully ported to Aarch64 and benefiting from scale. "With its debut on the TOP500, the 125,000-core Astra supercomputer at New Mexico’s Sandia Labs uses Cavium ThunderX2 chips to mark Arm’s entry into the petascale world. In Japan, the Fujitsu A64FX Arm-based CPU in the pending Fugaku supercomputer has been optimized to achieve high-level, real-world application performance, anticipating up to one hundred times the application execution performance of the K computer. K was the first computer to top 10 petaflops in 2011." Watch the video: https://wp.me/p3RLHQ-lIT Learn more: https://www.linaro.org/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Versal Premium ACAP for Network and Cloud Acceleration

inside-BigData.com

Today Xilinx announced Versal Premium, the third series in the Versal ACAP portfolio. The Versal Premium series features highly integrated, networked and power-optimized cores and the industry’s highest bandwidth and compute density on an adaptable platform. Versal Premium is designed for the highest bandwidth networks operating in thermally and spatially constrained environments, as well as for cloud providers who need scalable, adaptable application acceleration. Versal is the industry’s first adaptive compute acceleration platform (ACAP), a revolutionary new category of heterogeneous compute devices with capabilities that far exceed those of conventional silicon architectures. Developed on TSMC’s 7-nanometer process technology, Versal Premium combines software programmability with dynamically configurable hardware acceleration and pre-engineered connectivity and security features to enable a faster time-to- market. The Versal Premium series delivers up to 3X higher throughput compared to current generation FPGAs, with built-in Ethernet, Interlaken, and cryptographic engines that enable fast and secure networks. The series doubles the compute density of currently deployed mainstream FPGAs and provides the adaptability to keep pace with increasingly diverse and evolving cloud and networking workloads. Learn more: https://insidehpc.com/2020/03/xilinx-announces-versal-premium-acap-for-network-and-cloud-acceleration/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Zettar: Moving Massive Amounts of Data across Any Distance Efficiently

inside-BigData.com

In this video from the Rice Oil & Gas Conference, Chin Fang from Zettar presents: Moving Massive Amounts of Data across Any Distance Efficiently. The objective of this talk is to present two on-going projects aiming at improving and ensuring highly efficient bulk transferring or streaming of massive amounts of data over digital connections across any distance. It examines the current state of the art, a few very common misconceptions, the differences among the three major type of data movement solutions, a current initiative attempting to improve the data movement efficiency from the ground up, and another multi-stage project that shows how to conduct long distance large scale data movement at speed and scale internationally. Both projects have real world motivations, e.g. the ambitious data transfer requirements of Linac Coherent Light Source II (LCLS-II) [1], a premier preparation project of the U.S. DOE Exascale Computing Initiative (ECI) [2]. Their immediate goals are described and explained, together with the solution used for each. Findings and early results are reported. Possible future works are outlined. Watch the video: https://wp.me/p3RLHQ-lBX Learn more: https://www.zettar.com/ and https://rice2020oghpc.rice.edu/program-2/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Scaling TCO in a Post Moore's Era

inside-BigData.com

In this deck from the Rice Oil & Gas Conference, Bradley McCredie from AMD presents: Scaling TCO in a Post Moore's Law Era. "While foundries bravely drive forward to overcome the technical and economic challenges posed by scaling to 5nm and beyond, Moore’s law alone can provide only a fraction of the performance / watt and performance / dollar gains needed to satisfy the demands of today’s high performance computing and artificial intelligence applications. To close the gap, multiple strategies are required. First, new levels of innovation and design efficiency will supplement technology gains to continue to deliver meaningful improvements in SoC performance. Second, heterogenous compute architectures will create x-factor increases of performance efficiency for the most critical applications. Finally, open software frameworks, APIs, and toolsets will enable broad ecosystems of application level innovation." Watch the video: Learn more: http://amd.com and https://rice2020oghpc.rice.edu/program-2/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

CUDA-Python and RAPIDS for blazing fast scientific computing

inside-BigData.com

In this deck from the ECSS Symposium, Abe Stern from NVIDIA presents: CUDA-Python and RAPIDS for blazing fast scientific computing. "We will introduce Numba and RAPIDS for GPU programming in Python. Numba allows us to write just-in-time compiled CUDA code in Python, giving us easy access to the power of GPUs from a powerful high-level language. RAPIDS is a suite of tools with a Python interface for machine learning and dataframe operations. Together, Numba and RAPIDS represent a potent set of tools for rapid prototyping, development, and analysis for scientific computing. We will cover the basics of each library and go over simple examples to get users started. Finally, we will briefly highlight several other relevant libraries for GPU programming." Watch the video: https://wp.me/p3RLHQ-lvu Learn more: https://developer.nvidia.com/rapids and https://www.xsede.org/for-users/ecss/ecss-symposium Sign up for our insideHPC Newsletter: http://insidehp.com/newsletter

Introducing HPC with a Raspberry Pi Cluster

inside-BigData.com

In this deck from FOSDEM 2020, Colin Sauze from Aberystwyth University describes the development of a RaspberryPi cluster for teaching an introduction to HPC. "The motivation for this was to overcome four key problems faced by new HPC users: * The availability of a real HPC system and the effect running training courses can have on the real system, conversely the availability of spare resources on the real system can cause problems for the training course. * A fear of using a large and expensive HPC system for the first time and worries that doing something wrong might damage the system. * That HPC systems are very abstract systems sitting in data centres that users never see, it is difficult for them to understand exactly what it is they are using. * That new users fail to understand resource limitations, in part because of the vast resources in modern HPC systems a lot of mistakes can be made before running out of resources. A more resource constrained system makes it easier to understand this. The talk will also discuss some of the technical challenges in deploying an HPC environment to a Raspberry Pi and attempts to keep that environment as close to a "real" HPC as possible. The issue to trying to automate the installation process will also be covered." Learn more: https://github.com/colinsauze/pi_cluster and https://fosdem.org/2020/schedule/events/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Overview of HPC Interconnects

inside-BigData.com

In this deck from ATPESC 2019, Ken Raffenetti from Argonne presents an overview of HPC interconnects. "The Argonne Training Program on Extreme-Scale Computing (ATPESC) provides intensive, two-week training on the key skills, approaches, and tools to design, implement, and execute computational science and engineering applications on current high-end computing systems and the leadership-class computing systems of the future." Watch the video: https://wp.me/p3RLHQ-luc Learn more: https://extremecomputingtraining.anl.gov/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

More from inside-BigData.com (20)

Major Market Shifts in IT

Preparing to program Aurora at Exascale - Early experiences and future direct...

Transforming Private 5G Networks

The Incorporation of Machine Learning into Scientific Simulations at Lawrence...

How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...

Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...

HPC Impact: EDA Telemetry Neural Networks

Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring

Machine Learning for Weather Forecasts

HPC AI Advisory Council Update

Fugaku Supercomputer joins fight against COVID-19

Energy Efficient Computing using Dynamic Tuning

HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD

State of ARM-based HPC

Versal Premium ACAP for Network and Cloud Acceleration

Zettar: Moving Massive Amounts of Data across Any Distance Efficiently

Scaling TCO in a Post Moore's Era

CUDA-Python and RAPIDS for blazing fast scientific computing

Introducing HPC with a Raspberry Pi Cluster

Overview of HPC Interconnects

Recently uploaded

UiPath Test Automation using UiPath Test Suite series, part 4

DianaGray10

Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap. The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies. Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques What will you get from this session? 1. Insights into SAP testing best practices 2. Heatmap utilization for testing 3. Optimization of testing processes 4. Demo Topics covered: Execution from the test manager Orchestrator execution result Defect reporting SAP heatmap example with demo Speaker: Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

DevOps and Testing slides at DASA Connect

Kari Kakkonen

Assuring Contact Center Experiences for Your Customers With ThousandEyes

ThousandEyes

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

FIDO Alliance

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

FIDO Alliance

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

Connector Corner: Automate dynamic content and events by pushing a button

DianaGray10

Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to: Create a campaign using Mailchimp with merge tags/fields Send an interactive Slack channel message (using buttons) Have the message received by managers and peers along with a test email for review But there’s more: In a second workflow supporting the same use case, you’ll see: Your campaign sent to target colleagues for approval If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team But—if the “Reject” button is pushed, colleagues will be alerted via Slack message Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors. And... Speakers: Akshay Agnihotri, Product Manager Charlie Greenberg, Host

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

Product School

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

Product School

Elevating Tactical DDD Patterns Through Object Calisthenics

Dorra BARTAGUIZ

After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!

Key Trends Shaping the Future of Infrastructure.pdf

Cheryl Hung

Neuro-symbolic is not enough, we need neuro-*semantic*

Frank van Harmelen

Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”. All of this illustrated with link prediction over knowledge graphs, but the argument is general.

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

Inflectra

In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring. Learn about: • The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks. • Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective. • Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification. • Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process. Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Recently uploaded (20)

UiPath Test Automation using UiPath Test Suite series, part 4

DevOps and Testing slides at DASA Connect

Assuring Contact Center Experiences for Your Customers With ThousandEyes

FIDO Alliance Osaka Seminar: FIDO Security Aspects.pdf

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

GraphRAG is All You need? LLM & Knowledge Graph

Leading Change strategies and insights for effective change management pdf 1.pdf

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

Connector Corner: Automate dynamic content and events by pushing a button

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

Elevating Tactical DDD Patterns Through Object Calisthenics

Key Trends Shaping the Future of Infrastructure.pdf

Neuro-symbolic is not enough, we need neuro-*semantic*

The Art of the Pitch: WordPress Relationships and Sales

PCI PIN Basics Webinar from the Controlcase Team

Essentials of Automations: Optimizing FME Workflows with Parameters

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

State of ICS and IoT Cyber Threat Landscape Report 2024 preview