This document summarizes a presentation by IDC on big data and high performance data analysis (HPDA). It defines HPDA as combining data-intensive simulation and analytics tasks that require high-performance computing resources. The document outlines several major use cases for HPDA, including fraud detection, health care, and customer analytics. It also profiles specific examples like PayPal's use of HPC for fraud detection and GEICO's pre-calculation of insurance quotes. The document forecasts rapid growth in the HPDA market and notes that new technologies will be required to handle different types of workloads like graph analysis.
Earl Joseph from IDC presented this deck at the 2016 HPC User Forum in Austin.
Watch the video presentation: http://insidehpc.com/2016/09/60420/
Learn more: http://hpcuserforum.com
Addison Snell from Intersect360 Research presented this deck at the Stanford HPC Conference.
"Intersect360 Research returns with an annual deep dive into the trends, technologies and usage models that will be propelling the HPC community through 2017 and beyond. Emerging areas of focus and opportunities to expand will be explored along with insightful observations needed to support measurably positive decision making within your operations."
Watch the video: http://wp.me/p3RLHQ-gnJ
Learn more: http://intersect360research.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from ISC 2015, Earl Joseph and Bob Sorensen from IDC provide an HPC Market Update.
Watch the video presentation: http://wp.me/p3RLHQ-emz
Sign up for our insideHPC Newsletter: http://insideHPC.com/newsletter
In this video from the IDC Breakfast Briefing at ISC'13, Earl Joseph presents: HPC Trends.
View the presentation video:
http://insidehpc.com/?p=38402
Check out more talks from the show at our ISC'13 Video Gallery: http://insidehpc.com/isc13-video-gallery/
In this deck from Switzerland HPC Conference, Michael Feldman from TOP500.org presents an annual deep dive into the trends, technologies and usage models that will be propelling the HPC community through 2017 and beyond.
"Emerging areas of focus and opportunities to expand will be explored along with insightful observations needed to support measurably positive decision making within your operations."
Intersect360 Top of All Things in HPC Snapshot Analysisinside-BigData.com
The purpose of this report is to provide the top suppliers, packages, or categories within specific segments.
Learn more: http://intersect360research.com/industry/reports.php?id=144
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Chris Willard from Intersect360 Research gave this talk at the Stanford HPC Conference.
"Intersect360 Research returns with an annual deep dive into the trends, technologies and usage models that will be propelling the HPC community through 2018 and beyond. Emerging areas of focus and opportunities to expand will be explored along with insightful observations needed to support measurably positive decision making within your operations."
Intersect360 Research's deep knowledge of HPC, coupled with strong marketing and consulting expertise, results in actionable intelligence for the HPC industry - insights and advice that allow customers to make decisions that are measurably positive to their business. The company's end-user-focused research is inclusive from both a technology perspective and a usage standpoint, allowing Intersect360 Research to provide its clients with total market models that include both traditional and emerging HPC applications.
Watch the video: https://insidehpc.com/category/hpc-software/cloud-hpc/
Learn more: http://www.intersect360.com/
and
http://hpcadivosorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this video from the HPC User Forum in Milwaukee, Earl Joseph and Steve Conway from Hyperion Research present and update on HPC, AI, and Storage markets.
"Hyperion Research forecasts that the worldwide HPC server-based AI market will expand at a 29.5% CAGR to reach more than $1.26 billion in 2021, up more than three-fold from $346 million in 2016. We define the HPC AI market as a subset of the high-performance data analysis (HPDA) market that includes machine learning, deep learning, and other AI workloads running on HPC servers."
Learn more: http://hpcathyperion.com/
and
http://hpcuserforum.com
Earl Joseph from IDC presented this deck at the 2016 HPC User Forum in Austin.
Watch the video presentation: http://insidehpc.com/2016/09/60420/
Learn more: http://hpcuserforum.com
Addison Snell from Intersect360 Research presented this deck at the Stanford HPC Conference.
"Intersect360 Research returns with an annual deep dive into the trends, technologies and usage models that will be propelling the HPC community through 2017 and beyond. Emerging areas of focus and opportunities to expand will be explored along with insightful observations needed to support measurably positive decision making within your operations."
Watch the video: http://wp.me/p3RLHQ-gnJ
Learn more: http://intersect360research.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from ISC 2015, Earl Joseph and Bob Sorensen from IDC provide an HPC Market Update.
Watch the video presentation: http://wp.me/p3RLHQ-emz
Sign up for our insideHPC Newsletter: http://insideHPC.com/newsletter
In this video from the IDC Breakfast Briefing at ISC'13, Earl Joseph presents: HPC Trends.
View the presentation video:
http://insidehpc.com/?p=38402
Check out more talks from the show at our ISC'13 Video Gallery: http://insidehpc.com/isc13-video-gallery/
In this deck from Switzerland HPC Conference, Michael Feldman from TOP500.org presents an annual deep dive into the trends, technologies and usage models that will be propelling the HPC community through 2017 and beyond.
"Emerging areas of focus and opportunities to expand will be explored along with insightful observations needed to support measurably positive decision making within your operations."
Intersect360 Top of All Things in HPC Snapshot Analysisinside-BigData.com
The purpose of this report is to provide the top suppliers, packages, or categories within specific segments.
Learn more: http://intersect360research.com/industry/reports.php?id=144
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Chris Willard from Intersect360 Research gave this talk at the Stanford HPC Conference.
"Intersect360 Research returns with an annual deep dive into the trends, technologies and usage models that will be propelling the HPC community through 2018 and beyond. Emerging areas of focus and opportunities to expand will be explored along with insightful observations needed to support measurably positive decision making within your operations."
Intersect360 Research's deep knowledge of HPC, coupled with strong marketing and consulting expertise, results in actionable intelligence for the HPC industry - insights and advice that allow customers to make decisions that are measurably positive to their business. The company's end-user-focused research is inclusive from both a technology perspective and a usage standpoint, allowing Intersect360 Research to provide its clients with total market models that include both traditional and emerging HPC applications.
Watch the video: https://insidehpc.com/category/hpc-software/cloud-hpc/
Learn more: http://www.intersect360.com/
and
http://hpcadivosorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this video from the HPC User Forum in Milwaukee, Earl Joseph and Steve Conway from Hyperion Research present and update on HPC, AI, and Storage markets.
"Hyperion Research forecasts that the worldwide HPC server-based AI market will expand at a 29.5% CAGR to reach more than $1.26 billion in 2021, up more than three-fold from $346 million in 2016. We define the HPC AI market as a subset of the high-performance data analysis (HPDA) market that includes machine learning, deep learning, and other AI workloads running on HPC servers."
Learn more: http://hpcathyperion.com/
and
http://hpcuserforum.com
"This talk will discuss why it is important to measure High Performance Computing, and how to do so. The talk covers measuring performance, both technical (e.g., benchmarks) and non-technical (e.g., utilization); measuring the cost of HPC, from the simple beginnings to the complexity of Total Cost of Ownership (TCO) and beyond; and finally, the daunting world of measuring value, including the dreaded Return on Investment (ROI) and other metrics. The talk is based on NAG HPC consulting experiences with a range of industry HPC users and others. This is not a sales talk, nor a highly technical talk. It should be readily understood by anyone involved in using or managing HPC technology."
Watch the video presentation: http://wp.me/p3RLHQ-fJm
Learn more: http://nag.com
and
http://hpcuserforum.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this video from the HPC User Forum in Tucson, Earl Joseph from Hyperion Research presents an HPC Market Update.
"Hyperion Research is the new name for the former IDC high performance computing (HPC) analyst team. As Hyperion Research, we continue all the worldwide activities that spawned the world’s most respected HPC industry analyst group. For more than 25 years, we’ve helped IT professionals, business executives, and the investment community make fact-based decisions on technology purchases and business strategy."
Watch the video: https://wp.me/p3RLHQ-iqk
Learn more: https://hyperionresearch.com/
and
http://hpcuserforum.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
High Performance Data Analysis (HPDA): HPC - Big Data Convergenceinside-BigData.com
In this video from the HPC User Forum in Santa Fe, Steve Conway from Hyperion Research presents: High Performance Data Analysis (HPDA): HPC - Big Data Convergence.
"To date, most data-intensive HPC jobs in the government, academic and industrial sectors have involved the modeling and simulation of complex physical and quasi-physical systems. The systems range from product designs for cars, planes, golf clubs and pharmaceuticals, to subatomic particles, global weather and climate patterns, and the cosmos itself. But from the start of the supercomputer era in the 1960s — and even earlier —an important subset of HPC jobs has involved analytics — attempts to uncover useful information and patterns in the data itself. Cryptography, one of the original scientific-technical computing applications, falls predominantly into this category."
Watch the video: http://wp.me/p3RLHQ-gHm
Learn more: http://hpcuserforum.com
Shahin Khan from OrionX presented this deck at the Stanford HPC Conference.
"From BitCoins and AltCoins to Design Thinking, Autonomous tech and the changing nature of jobs, IoT and cyber risk, and the impact of application architecture on cloud computing, we’ll touch on some of the hottest technologies in 2017 that are changing the world and how HPC will be the engine that drives it."
Watch the video: http://insidehpc.com/2017/02/shahin-khan-presents-hot-technology-topics-2017/
Learn more: http://orionx.net
and
http://www.hpcadvisorycouncil.com/events/2017/stanford-workshop/
Sign up for our insideHPC Newsletter: http:/insidehpc.com/newsletter
Application Profiling at the HPCAC High Performance Centerinside-BigData.com
Pak Lui from the HPC Advisory Council presented this deck at the 2017 Stanford HPC Conference.
"To achieve good scalability performance on the HPC scientific applications typically involves good understanding of the workload though performing profile analysis, and comparing behaviors of using different hardware which pinpoint bottlenecks in different areas of the HPC cluster. In this session, a selection of HPC applications will be shown to demonstrate various methods of profiling and analysis to determine the bottleneck, and the effectiveness of the tuning to improve on the application performance from tests conducted at the HPC Advisory Council High Performance Center."
Watch the video presentation: http://wp.me/p3RLHQ-gpY
Learn more: http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from Radio Free HPC, Dan Olds and Shahin Khan from OrionX describe their new AI Survey.
"OrionX Research has completed one the most comprehensive surveys to date of Artificial Intelligence, Machine Learning, and Deep Learning. With over 300 respondents in North America, representing 13 industries, our model indicates a confidence level of 95% and a margin of error of 6%. Covering 144 questions/data points, it provides a comprehensive view of what customers are doing and planning to do with AI/ML/DL."
Listen to the podcast: https://insidehpc.com/2017/09/new-orionx-survey-insights-artificial-intelligence/
Learn more: http://orionx.net/ai
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from the HPC User Forum in Milwaukee, Bob Sorensen from Hyperion Research describes an ongoing study on the Development Trends of Next-Generation Supercomputers.
Project Requirements:
* Gather information on pre-exascale and exascale systems today and through 2028
* Concentrate on major HPC developer countries: US, China, EU, Japan, others?
* Build database of technical information on the research and development efforts on these next-generation machines
* Collect information on the flow of funding (amount from the country to the companies, etc.)
Hyperion Research is the new name for the former IDC high performance computing (HPC) analyst team. As Hyperion Research, we continue all the worldwide activities that spawned the world’s most respected HPC industry analyst group. For more than 25 years, we’ve helped IT professionals, business executives, and the investment community make fact-based decisions on technology purchases and business strategy.
Watch the video: https://wp.me/p3RLHQ-hlY
Learn more: http://www.hpcathyperion.com/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from the Stanford HPC Conference, Shahin Khan from OrionX presents: Outlook on Hot Technologies.
"We will review OrionX's predictions for 2018, the technologies that are changing the world (Iot, Blockchain, Quantum Computing, AI ...) and how HPC will be the engine that drives it."
Watch the video: https://youtu.be/Qyy1NgxN4b8
Learn more: http://orionx.net
and
http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
BIG Data & Hadoop Applications in LogisticsSkillspeed
Explore the applications of BIG Data & Hadoop in Logistics via Skillspeed.
BIG Data & Hadoop in Logistics is a key differentiator, especially in terms of optimizing back-end operations. They are used by companies for delivery optimization, demand & inventory forecasting and simplifying distribution networks.
To get more details regarding BIG Data & Hadoop, please visit - www.SkillSpeed.com
Check out this white paper from eInfochips which showcases how energy and utility providers can unlock potential service opportunities using our predictive analytics solution across all stages of the business cycle. Major utility players are set to roll out millions of smart meters with the aim of generating actionable insights even though as per the industry’s own admission, any serious effort toward monetization is being offset by a lack of core IT capabilities, especially in big data technology. Capturing proactive intelligence on consumer behavior is the way to go. In this white paper, eInfochips demonstrates how utility players can predict demand response, generation response and create new revenue models around coincidental peak demands, smart expenditure modeling and other forms of end user data.
Abivin - Big Data Analytics & OptimizationLong Pham
We presenting here our Vision of Big Data for Logistics for Companies in Southeast Asia. We are building a product Abivin vRoute that could help to manage fleets of vehicles, perform route optimization and GPS trackings. The future is bright to us with a large market in Southeast Asia and many other problems to solve to benefit the society.
Visit this link to complete the quiz - https://mix.office.com/watch/ays9xktksvjb The Data Asset Introduction - Databases, Business Intelligence, Analytics, Big Data, and Competitive Advantage
Needle in the Haystack by Anshul Vikram Pandey at QuantCon 2016Quantopian
The amount of text data (news articles, blogs, social media etc.) on the web is increasing at a staggering rate. However, the amount of irrelevant information or noise on the web is increasing at a much higher rate than action-able information that can generate alpha. It is becoming increasingly difficult to mine for actionable stories on the web using standard, out of the box language processing techniques and libraries. Given that the performance, robustness and reliability of all data-centric models are directly dependent on the quality of the data, noise reduction becomes one of the most important steps in the data science pipeline. Thanks to the recent research advancements in the field of big data, deep learning and natural language processing technologies, we are now able to mine for actionable stories in millions of information pieces and hundreds of terabytes of data.
In this talk, we will highlight various approaches and technologies we employ as part of the noise cancellation mechanism at Accern. We will also compare the performance of trading strategies that use social analytics derived using standard versus sophisticated noise cancellation techniques, as well as those that utilize other advanced metrics.
Big Data: Smart Technologies Provide Big OpportunitiesNAED_Org
Big data has garnered big-time buzz as an effective means to optimize business and measure success. This concise report provides an introduction to the elements of big data and how smart technologies are playing a big role in the information game.
That's not a metric! Data for cloud-native successGordon Haff
“Without data, you’re just another person with an opinion.” W. Edwards Deming was talking about statistical quality control in manufacturing but he could equally have been referring to managing modern iterative and automated software deployment pipelines and cloud-native infrastructure. Certainly there's a wealth of open source tools to capture and visualize data. However, a data strategy isn’t solely or even mostly about drawing up a long list of technical measurements and instrumenting software to capture everything.
It's crucial to distinguish between metrics that relate software initiatives to positive business outcomes, the alerts needed to respond to problems now, and the data required for root cause analysis or to optimize processes over time. All data is not equal. And most data is not a metric for measuring success.
In this deck from the HPC User Forum in Tucson, Earl Joseph from IDC presents: 2016 IDC HPC Market Update.
Watch the video presentation: http://wp.me/p3RLHQ-fbt
Learn more: http://hpcuserforum.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Hagen Toennies from Gaikai Inc. presented this deck at the 2017 HPC Advisory Council Stanford Conference.
"In this talk we will present how we enable distributed, Unix style programming using Docker and Apache Kafka. We will show how we can take the famous Unix Pipe Pattern and apply it to a Distributed Computing System. We will demonstrate the development of two simple applications with the focus on "Do One Thing and Do It Well." Afterwards we demonstrate how we make these two programs work to together using Apache Kafka. By encapsulating our applications in containers we will also show how that enables us to go from the limited resources of a development machine to cluster of computers in a data center without changing our applications or containers."
Watch the video: http://wp.me/p3RLHQ-goG
Learn more: http://www.hpcadvisorycouncil.com/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
"This talk will discuss why it is important to measure High Performance Computing, and how to do so. The talk covers measuring performance, both technical (e.g., benchmarks) and non-technical (e.g., utilization); measuring the cost of HPC, from the simple beginnings to the complexity of Total Cost of Ownership (TCO) and beyond; and finally, the daunting world of measuring value, including the dreaded Return on Investment (ROI) and other metrics. The talk is based on NAG HPC consulting experiences with a range of industry HPC users and others. This is not a sales talk, nor a highly technical talk. It should be readily understood by anyone involved in using or managing HPC technology."
Watch the video presentation: http://wp.me/p3RLHQ-fJm
Learn more: http://nag.com
and
http://hpcuserforum.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this video from the HPC User Forum in Tucson, Earl Joseph from Hyperion Research presents an HPC Market Update.
"Hyperion Research is the new name for the former IDC high performance computing (HPC) analyst team. As Hyperion Research, we continue all the worldwide activities that spawned the world’s most respected HPC industry analyst group. For more than 25 years, we’ve helped IT professionals, business executives, and the investment community make fact-based decisions on technology purchases and business strategy."
Watch the video: https://wp.me/p3RLHQ-iqk
Learn more: https://hyperionresearch.com/
and
http://hpcuserforum.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
High Performance Data Analysis (HPDA): HPC - Big Data Convergenceinside-BigData.com
In this video from the HPC User Forum in Santa Fe, Steve Conway from Hyperion Research presents: High Performance Data Analysis (HPDA): HPC - Big Data Convergence.
"To date, most data-intensive HPC jobs in the government, academic and industrial sectors have involved the modeling and simulation of complex physical and quasi-physical systems. The systems range from product designs for cars, planes, golf clubs and pharmaceuticals, to subatomic particles, global weather and climate patterns, and the cosmos itself. But from the start of the supercomputer era in the 1960s — and even earlier —an important subset of HPC jobs has involved analytics — attempts to uncover useful information and patterns in the data itself. Cryptography, one of the original scientific-technical computing applications, falls predominantly into this category."
Watch the video: http://wp.me/p3RLHQ-gHm
Learn more: http://hpcuserforum.com
Shahin Khan from OrionX presented this deck at the Stanford HPC Conference.
"From BitCoins and AltCoins to Design Thinking, Autonomous tech and the changing nature of jobs, IoT and cyber risk, and the impact of application architecture on cloud computing, we’ll touch on some of the hottest technologies in 2017 that are changing the world and how HPC will be the engine that drives it."
Watch the video: http://insidehpc.com/2017/02/shahin-khan-presents-hot-technology-topics-2017/
Learn more: http://orionx.net
and
http://www.hpcadvisorycouncil.com/events/2017/stanford-workshop/
Sign up for our insideHPC Newsletter: http:/insidehpc.com/newsletter
Application Profiling at the HPCAC High Performance Centerinside-BigData.com
Pak Lui from the HPC Advisory Council presented this deck at the 2017 Stanford HPC Conference.
"To achieve good scalability performance on the HPC scientific applications typically involves good understanding of the workload though performing profile analysis, and comparing behaviors of using different hardware which pinpoint bottlenecks in different areas of the HPC cluster. In this session, a selection of HPC applications will be shown to demonstrate various methods of profiling and analysis to determine the bottleneck, and the effectiveness of the tuning to improve on the application performance from tests conducted at the HPC Advisory Council High Performance Center."
Watch the video presentation: http://wp.me/p3RLHQ-gpY
Learn more: http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from Radio Free HPC, Dan Olds and Shahin Khan from OrionX describe their new AI Survey.
"OrionX Research has completed one the most comprehensive surveys to date of Artificial Intelligence, Machine Learning, and Deep Learning. With over 300 respondents in North America, representing 13 industries, our model indicates a confidence level of 95% and a margin of error of 6%. Covering 144 questions/data points, it provides a comprehensive view of what customers are doing and planning to do with AI/ML/DL."
Listen to the podcast: https://insidehpc.com/2017/09/new-orionx-survey-insights-artificial-intelligence/
Learn more: http://orionx.net/ai
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from the HPC User Forum in Milwaukee, Bob Sorensen from Hyperion Research describes an ongoing study on the Development Trends of Next-Generation Supercomputers.
Project Requirements:
* Gather information on pre-exascale and exascale systems today and through 2028
* Concentrate on major HPC developer countries: US, China, EU, Japan, others?
* Build database of technical information on the research and development efforts on these next-generation machines
* Collect information on the flow of funding (amount from the country to the companies, etc.)
Hyperion Research is the new name for the former IDC high performance computing (HPC) analyst team. As Hyperion Research, we continue all the worldwide activities that spawned the world’s most respected HPC industry analyst group. For more than 25 years, we’ve helped IT professionals, business executives, and the investment community make fact-based decisions on technology purchases and business strategy.
Watch the video: https://wp.me/p3RLHQ-hlY
Learn more: http://www.hpcathyperion.com/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from the Stanford HPC Conference, Shahin Khan from OrionX presents: Outlook on Hot Technologies.
"We will review OrionX's predictions for 2018, the technologies that are changing the world (Iot, Blockchain, Quantum Computing, AI ...) and how HPC will be the engine that drives it."
Watch the video: https://youtu.be/Qyy1NgxN4b8
Learn more: http://orionx.net
and
http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
BIG Data & Hadoop Applications in LogisticsSkillspeed
Explore the applications of BIG Data & Hadoop in Logistics via Skillspeed.
BIG Data & Hadoop in Logistics is a key differentiator, especially in terms of optimizing back-end operations. They are used by companies for delivery optimization, demand & inventory forecasting and simplifying distribution networks.
To get more details regarding BIG Data & Hadoop, please visit - www.SkillSpeed.com
Check out this white paper from eInfochips which showcases how energy and utility providers can unlock potential service opportunities using our predictive analytics solution across all stages of the business cycle. Major utility players are set to roll out millions of smart meters with the aim of generating actionable insights even though as per the industry’s own admission, any serious effort toward monetization is being offset by a lack of core IT capabilities, especially in big data technology. Capturing proactive intelligence on consumer behavior is the way to go. In this white paper, eInfochips demonstrates how utility players can predict demand response, generation response and create new revenue models around coincidental peak demands, smart expenditure modeling and other forms of end user data.
Abivin - Big Data Analytics & OptimizationLong Pham
We presenting here our Vision of Big Data for Logistics for Companies in Southeast Asia. We are building a product Abivin vRoute that could help to manage fleets of vehicles, perform route optimization and GPS trackings. The future is bright to us with a large market in Southeast Asia and many other problems to solve to benefit the society.
Visit this link to complete the quiz - https://mix.office.com/watch/ays9xktksvjb The Data Asset Introduction - Databases, Business Intelligence, Analytics, Big Data, and Competitive Advantage
Needle in the Haystack by Anshul Vikram Pandey at QuantCon 2016Quantopian
The amount of text data (news articles, blogs, social media etc.) on the web is increasing at a staggering rate. However, the amount of irrelevant information or noise on the web is increasing at a much higher rate than action-able information that can generate alpha. It is becoming increasingly difficult to mine for actionable stories on the web using standard, out of the box language processing techniques and libraries. Given that the performance, robustness and reliability of all data-centric models are directly dependent on the quality of the data, noise reduction becomes one of the most important steps in the data science pipeline. Thanks to the recent research advancements in the field of big data, deep learning and natural language processing technologies, we are now able to mine for actionable stories in millions of information pieces and hundreds of terabytes of data.
In this talk, we will highlight various approaches and technologies we employ as part of the noise cancellation mechanism at Accern. We will also compare the performance of trading strategies that use social analytics derived using standard versus sophisticated noise cancellation techniques, as well as those that utilize other advanced metrics.
Big Data: Smart Technologies Provide Big OpportunitiesNAED_Org
Big data has garnered big-time buzz as an effective means to optimize business and measure success. This concise report provides an introduction to the elements of big data and how smart technologies are playing a big role in the information game.
That's not a metric! Data for cloud-native successGordon Haff
“Without data, you’re just another person with an opinion.” W. Edwards Deming was talking about statistical quality control in manufacturing but he could equally have been referring to managing modern iterative and automated software deployment pipelines and cloud-native infrastructure. Certainly there's a wealth of open source tools to capture and visualize data. However, a data strategy isn’t solely or even mostly about drawing up a long list of technical measurements and instrumenting software to capture everything.
It's crucial to distinguish between metrics that relate software initiatives to positive business outcomes, the alerts needed to respond to problems now, and the data required for root cause analysis or to optimize processes over time. All data is not equal. And most data is not a metric for measuring success.
In this deck from the HPC User Forum in Tucson, Earl Joseph from IDC presents: 2016 IDC HPC Market Update.
Watch the video presentation: http://wp.me/p3RLHQ-fbt
Learn more: http://hpcuserforum.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Hagen Toennies from Gaikai Inc. presented this deck at the 2017 HPC Advisory Council Stanford Conference.
"In this talk we will present how we enable distributed, Unix style programming using Docker and Apache Kafka. We will show how we can take the famous Unix Pipe Pattern and apply it to a Distributed Computing System. We will demonstrate the development of two simple applications with the focus on "Do One Thing and Do It Well." Afterwards we demonstrate how we make these two programs work to together using Apache Kafka. By encapsulating our applications in containers we will also show how that enables us to go from the limited resources of a development machine to cluster of computers in a data center without changing our applications or containers."
Watch the video: http://wp.me/p3RLHQ-goG
Learn more: http://www.hpcadvisorycouncil.com/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
WSO2Con USA 2017: Scalable Real-time Complex Event Processing at UberWSO2
The Marketplace data team at Uber has built a scalable complex event processing platform to solve many challenging real-time data needs for various Uber products. This platform has been in production for more than a year and supports over 100 real-time data use cases with a team of 3. In this talk, we will share the detail of the design and our experience, and how we employ Siddhi, Kafka and Samza at scale.
In this deck from the 2014 HPC User Forum in Seattle, Ed Turkel from HP presents an update on HPC at HP.
Watch the video presentation: http://wp.me/p3RLHQ-d9X
Business Wizard Of The Year : Mr. AMAR BABU,M.D.-LENOVO INDIAVARINDIA
Business Wizard Of The Year : Mr. AMAR BABU,M.D.-LENOVO INDIA: Award given by Shri . SUNIL SONI,IAS-Director General, BIS-Govt. of India and Shri . S.N.Tripathy, IAS- J.S, MSME-Govt. of India to Mr. AMAR BABU,M.D.-LENOVO INDIA , being received by his team member.....Go for More Details-------->http://www.varindia.com/star-nite-2014-business-wizard-of-the-year-amar-babu.html
Percy Tzelnic from Dell Technologies presented this deck at the HPC User Forum in Austin.
Watch the video presentation: http://insidehpc.com/2016/09/emc-in-hpc-the-journey-so-far-and-the-road-ahead/
Learn more: http://emc.com/
Frank Ham from Cascade Technologies presented this deck at the Stanford HPC Conference.
"A spin-off of the Center for Turbulence Research at Stanford University, Cascade Technologies grew out of a need to bridge between fundamental research from institutions like Stanford University and its application in industries. In a continual push to improve the operability and performance of combustion devices, high-fidelity simulation methods for turbulent combustion are emerging as critical elements in the design process. Multiphysics based methodologies can accurately predict mixing, study flame structure and stability, and even predict product and pollutant concentrations at design and off-design conditions."
Watch the video: http://insidehpc.com/2017/02/best-practices-large-scale-multiphysics/
Learn more: http://www.cascadetechnologies.com
and
http://www.hpcadvisorycouncil.com/events/2017/stanford-workshop/
Sign up for our insideHPC Newsletter: http:/insidehpc.com/newsletter
In this deck from the 2015 PBS Works User Group, Michael Thompson from Wayne State University presents: Maximizing HPC Compute Resources with Minimal Cost.
"As HPC resource requirements continue to increase, the need for finding economical solutions to handle the rising requirements increases as well. There are numerous ways to approach this challenge, each of which have varying return on investment (ROI); unfortunately, some options that involve a higher ROI are often unknown or overlooked. For example, leveraging existing equipment, adding new or used equipment, and handling uncommon peak usage dynamically through cloud solutions managed by a central job management system can prove to be highly available and resource rich, while remaining economical. In this presentation we will discuss how Wayne State University implemented a combination of these approaches to dramatically increase our compute resources for the equivalent cost of only a few new servers."
Learn more: http://www.pbsworks.com/pbsug/2015/agenda.aspx
Watch the video presentation: https://www.youtube.com/watch?v=mO98cr5NwME
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Learn more: http://www.pbsworks.com/pbsug/2015/agenda.aspx
Watch the video presentation: https://www.youtube.com/watch?v=NP4HfZm5e7w
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Un año más hemos presentado las Previsiones de IDC España para el año 2014.
Once again IDC Spain has presented our Predictions for 2014.
#IDCpredictions
AMD has been away from the HPC space for a while, but now they are coming back in a big way with an open software approach to GPU computing. The Radeon Open Compute Platform (ROCm) was born from the Boltzman Initiative announced last year at SC15. Now available on GitHub, the ROCm Platform bringing a rich foundation to advanced computing by better integrating the CPU and GPU to solve real-world problems.
"We are excited to present ROCm, the first open-source HPC/ultrascale-class platform for GPU computing that’s also programming-language independent. We are bringing the UNIX philosophy of choice, minimalism and modular software development to GPU computing. The new ROCm foundation lets you choose or even develop tools and a language run time for your application."
Watch the video presentation: http://wp.me/p3RLHQ-fJT
Learn more: https://radeonopencompute.github.io/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Modern Computing: Cloud, Distributed, & High Performanceinside-BigData.com
In this video, Dr. Umit Catalyurek from Georgia Institute of Technology presents: Modern Computing: Cloud, Distributed, & High Performance.
Ümit V. Çatalyürek is a Professor in the School of Computational Science and Engineering in the College of Computing at the Georgia Institute of Technology. He received his Ph.D. in 2000 from Bilkent University. He is a recipient of an NSF CAREER award and is the primary investigator of several awards from the Department of Energy, the National Institute of Health, and the National Science Foundation. He currently serves as an Associate Editor for Parallel Computing, and as an editorial board member for IEEE Transactions on Parallel and Distributed Computing, and the Journal of Parallel and Distributed Computing.
Learn more: http://www.bigdatau.org/data-science-seminars
Watch the video presentation: http://wp.me/p3RLHQ-ghU
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck, the Radio Free HPC team reviews the results from SC16 Student Cluster Competition.
Watch the video presentation: http://wp.me/p3RLHQ-g2G
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Damian Rouson and Alessandro Fanfarillo gave this Tutorial at the Stanford HPC Conference.
"This tutorial will present several features that the draft Fortran 2015 standard introduces to meet challenges that are expected to dominate massively parallel programming in the coming exascale era. The expected exascale challenges include higher hardware- and software-failure rates, increasing hardware heterogeneity, a proliferation of execution units, and deeper memory hierarchies."
Watch the video: http://insidehpc.com/2017/02/tutorial-towards-exascale-computing-fortran-2015/
Learn more: http://www.sourceryinstitute.org/store.html
and
http://www.hpcadvisorycouncil.com/events/2017/stanford-workshop/
Sign up for our insideHPC Newsletter: http:/insidehpc.com/newsletter
Don't Fall Into a Trap: How Business Continuity Management Can Help Data Brea...IBM Services
IT systems go down, important records are exposed, credit card numbers are compromised and identities are bought and sold on dark corners of the internet. The road to recovery can be a long one and can leave a mark some organizations have a hard time erasing in the long term. This Ebook describes how business continuity management can help.
Big Data brings big promise and also big challenges, the primary and most important one being the ability to deliver Value to business stakeholders who are not data scientists!
A l'occasion de l'eGov Innovation Day 2014 - DONNÉES DE L’ADMINISTRATION, UNE MINE (qui) D’OR(t) - Philippe Cudré-Mauroux présente Big Data et eGovernment.
seminar on Big Data Technology
report on big data technology
webinar on big data technology
topic on big data technology
ppt presentation on big data technology
Similar to IDC Perspectives on Big Data Outside of HPC (20)
In this deck from the Stanford HPC Conference, Shahin Khan from OrionX describes major market Shifts in IT.
"We will discuss the digital infrastructure of the future enterprise and the state of these trends."
"We work with clients on the impact of Digital Transformation (DX) on them, their customers, and their messages. Generally, they want to track, in one place, trends like IoT, 5G, AI, Blockchain, and Quantum Computing. And they want to know what these trends mean, how they affect each other, and when they demand action, and how to formulate and execute an effective plan. If that describes you, we can help."
Watch the video: https://wp.me/p3RLHQ-lPP
Learn more: http://orionx.net
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Preparing to program Aurora at Exascale - Early experiences and future direct...inside-BigData.com
In this deck from IWOCL / SYCLcon 2020, Hal Finkel from Argonne National Laboratory presents: Preparing to program Aurora at Exascale - Early experiences and future directions.
"Argonne National Laboratory’s Leadership Computing Facility will be home to Aurora, our first exascale supercomputer. Aurora promises to take scientific computing to a whole new level, and scientists and engineers from many different fields will take advantage of Aurora’s unprecedented computational capabilities to push the boundaries of human knowledge. In addition, Aurora’s support for advanced machine-learning and big-data computations will enable scientific workflows incorporating these techniques along with traditional HPC algorithms. Programming the state-of-the-art hardware in Aurora will be accomplished using state-of-the-art programming models. Some of these models, such as OpenMP, are long-established in the HPC ecosystem. Other models, such as Intel’s oneAPI, based on SYCL, are relatively-new models constructed with the benefit of significant experience. Many applications will not use these models directly, but rather, will use C++ abstraction libraries such as Kokkos or RAJA. Python will also be a common entry point to high-performance capabilities. As we look toward the future, features in the C++ standard itself will become increasingly relevant for accessing the extreme parallelism of exascale platforms.
This presentation will summarize the experiences of our team as we prepare for Aurora, exploring how to port applications to Aurora’s architecture and programming models, and distilling the challenges and best practices we’ve developed to date. oneAPI/SYCL and OpenMP are both critical models in these efforts, and while the ecosystem for Aurora has yet to mature, we’ve already had a great deal of success. Importantly, we are not passive recipients of programming models developed by others. Our team works not only with vendor-provided compilers and tools, but also develops improved open-source LLVM-based technologies that feed both open-source and vendor-provided capabilities. In addition, we actively participate in the standardization of OpenMP, SYCL, and C++. To conclude, I’ll share our thoughts on how these models can best develop in the future to support exascale-class systems."
Watch the video: https://wp.me/p3RLHQ-lPT
Learn more: https://www.iwocl.org/iwocl-2020/conference-program/
and
https://www.anl.gov/topic/aurora
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck, Greg Wahl from Advantech presents: Transforming Private 5G Networks.
Advantech Networks & Communications Group is driving innovation in next-generation network solutions with their High Performance Servers. We provide business critical hardware to the world's leading telecom and networking equipment manufacturers with both standard and customized products. Our High Performance Servers are highly configurable platforms designed to balance the best in x86 server-class processing performance with maximum I/O and offload density. The systems are cost effective, highly available and optimized to meet next generation networking and media processing needs.
“Advantech’s Networks and Communication Group has been both an innovator and trusted enabling partner in the telecommunications and network security markets for over a decade, designing and manufacturing products for OEMs that accelerate their network platform evolution and time to market.” Said Advantech Vice President of Networks & Communications Group, Ween Niu. “In the new IP Infrastructure era, we will be expanding our expertise in Software Defined Networking (SDN) and Network Function Virtualization (NFV), two of the essential conduits to 5G infrastructure agility making networks easier to install, secure, automate and manage in a cloud-based infrastructure.”
In addition to innovation in air interface technologies and architecture extensions, 5G will also need a new generation of network computing platforms to run the emerging software defined infrastructure, one that provides greater topology flexibility, essential to deliver on the promises of high availability, high coverage, low latency and high bandwidth connections. This will open up new parallel industry opportunities through dedicated 5G network slices reserved for specific industries dedicated to video traffic, augmented reality, IoT, connected cars etc. 5G unlocks many new doors and one of the keys to its enablement lies in the elasticity and flexibility of the underlying infrastructure.
Advantech’s corporate vision is to enable an intelligent planet. The company is a global leader in the fields of IoT intelligent systems and embedded platforms. To embrace the trends of IoT, big data, and artificial intelligence, Advantech promotes IoT hardware and software solutions with the Edge Intelligence WISE-PaaS core to assist business partners and clients in connecting their industrial chains. Advantech is also working with business partners to co-create business ecosystems that accelerate the goal of industrial intelligence."
Watch the video: https://wp.me/p3RLHQ-lPQ
* Company website: https://www.advantech.com/
* Solution page: https://www2.advantech.com/nc/newsletter/NCG/SKY/benefits.html
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...inside-BigData.com
In this deck from the Stanford HPC Conference, Katie Lewis from Lawrence Livermore National Laboratory presents: The Incorporation of Machine Learning into Scientific Simulations at Lawrence Livermore National Laboratory.
"Scientific simulations have driven computing at Lawrence Livermore National Laboratory (LLNL) for decades. During that time, we have seen significant changes in hardware, tools, and algorithms. Today, data science, including machine learning, is one of the fastest growing areas of computing, and LLNL is investing in hardware, applications, and algorithms in this space. While the use of simulations to focus and understand experiments is well accepted in our community, machine learning brings new challenges that need to be addressed. I will explore applications for machine learning in scientific simulations that are showing promising results and further investigation that is needed to better understand its usefulness."
Watch the video: https://youtu.be/NVwmvCWpZ6Y
Learn more: https://computing.llnl.gov/research-area/machine-learning
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...inside-BigData.com
In this deck from the Stanford HPC Conference, DK Panda from Ohio State University presents: How to Achieve High-Performance, Scalable and Distributed DNN Training on Modern HPC Systems?
"This talk will start with an overview of challenges being faced by the AI community to achieve high-performance, scalable and distributed DNN training on Modern HPC systems with both scale-up and scale-out strategies. After that, the talk will focus on a range of solutions being carried out in my group to address these challenges. The solutions will include: 1) MPI-driven Deep Learning, 2) Co-designing Deep Learning Stacks with High-Performance MPI, 3) Out-of- core DNN training, and 4) Hybrid (Data and Model) parallelism. Case studies to accelerate DNN training with popular frameworks like TensorFlow, PyTorch, MXNet and Caffe on modern HPC systems will be presented."
Watch the video: https://youtu.be/LeUNoKZVuwQ
Learn more: http://web.cse.ohio-state.edu/~panda.2/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...inside-BigData.com
In this deck from the Stanford HPC Conference, Nick Nystrom and Paola Buitrago provide an update from the Pittsburgh Supercomputing Center.
Nick Nystrom is Chief Scientist at the Pittsburgh Supercomputing Center (PSC). Nick is architect and PI for Bridges, PSC's flagship system that successfully pioneered the convergence of HPC, AI, and Big Data. He is also PI for the NIH Human Biomolecular Atlas Program’s HIVE Infrastructure Component and co-PI for projects that bring emerging AI technologies to research (Open Compass), apply machine learning to biomedical data for breast and lung cancer (Big Data for Better Health), and identify causal relationships in biomedical big data (the Center for Causal Discovery, an NIH Big Data to Knowledge Center of Excellence). His current research interests include hardware and software architecture, applications of machine learning to multimodal data (particularly for the life sciences) and to enhance simulation, and graph analytics.
Watch the video: https://youtu.be/LWEU1L1o7yY
Learn more: https://www.psc.edu/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from the Stanford HPC Conference, Ryan Quick from Providentia Worldwide describes how DNNs can be used to improve EDA simulation runs.
"Systems Intelligence relies on a variety of methods for providing insight into the core mechanisms for driving automated behavioral changes in self-healing command and control platforms. This talk reports on initial efforts with leveraging Semiconductor Electronic Design Automation (EDA) telemetry data from cross-domain sources including power, network, storage, nodes, and applications in neural networks as a driving method for insight into SI automation systems."
Watch the video: https://youtu.be/2WbR8tq-XbM
Learn more: http://www.providentiaworldwide.com/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoringinside-BigData.com
In this deck from the Stanford HPC Conference, Nicole Xu from Stanford University describes how she transformed a common jellyfish into a bionic creature that is part animal and part machine.
"Animal locomotion and bioinspiration have the potential to expand the performance capabilities of robots, but current implementations are limited. Mechanical soft robots leverage engineered materials and are highly controllable, but these biomimetic robots consume more power than corresponding animal counterparts. Biological soft robots from a bottom-up approach offer advantages such as speed and controllability but are limited to survival in cell media. Instead, biohybrid robots that comprise live animals and self- contained microelectronic systems leverage the animals’ own metabolism to reduce power constraints and body as an natural scaffold with damage tolerance. We demonstrate that by integrating onboard microelectronics into live jellyfish, we can enhance propulsion up to threefold, using only 10 mW of external power input to the microelectronics and at only a twofold increase in cost of transport to the animal. This robotic system uses 10 to 1000 times less external power per mass than existing swimming robots in literature and can be used in future applications for ocean monitoring to track environmental changes."
Watch the video: https://youtu.be/HrmJFyvInj8
Learn more: https://sanfrancisco.cbslocal.com/2020/02/05/stanford-research-project-common-jellyfish-bionic-sea-creatures/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from the Stanford HPC Conference, Peter Dueben from the European Centre for Medium-Range Weather Forecasts (ECMWF) presents: Machine Learning for Weather Forecasts.
"I will present recent studies that use deep learning to learn the equations of motion of the atmosphere, to emulate model components of weather forecast models and to enhance usability of weather forecasts. I will than talk about the main challenges for the application of deep learning in cutting-edge weather forecasts and suggest approaches to improve usability in the future."
Peter is contributing to the development and optimization of weather and climate models for modern supercomputers. He is focusing on a better understanding of model error and model uncertainty, on the use of reduced numerical precision that is optimised for a given level of model error, on global cloud- resolving simulations with ECMWF's forecast model, and the use of machine learning, and in particular deep learning, to improve the workflow and predictions. Peter has graduated in Physics and wrote his PhD thesis at the Max Planck Institute for Meteorology in Germany. He worked as Postdoc with Tim Palmer at the University of Oxford and has taken up a position as University Research Fellow of the Royal Society at the European Centre for Medium-Range Weather Forecasts (ECMWF) in 2017.
Watch the video: https://youtu.be/ks3fkRj8Iqc
Learn more: https://www.ecmwf.int/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck, Gilad Shainer from the HPC AI Advisory Council describes how this organization fosters innovation in the high performance computing community.
"The HPC-AI Advisory Council’s mission is to bridge the gap between high-performance computing (HPC) and Artificial Intelligence (AI) use and its potential, bring the beneficial capabilities of HPC and AI to new users for better research, education, innovation and product manufacturing, bring users the expertise needed to operate HPC and AI systems, provide application designers with the tools needed to enable parallel computing, and to strengthen the qualification and integration of HPC and AI system products."
Watch the video: https://wp.me/p3RLHQ-lNz
Learn more: http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Today RIKEN in Japan announced that the Fugaku supercomputer will be made available for research projects aimed to combat COVID-19.
"Fugaku is currently being installed and is scheduled to be available to the public in 2021. However, faced with the devastating disaster unfolding before our eyes, RIKEN and MEXT decided to make a portion of the computational resources of Fugaku available for COVID-19-related projects ahead of schedule while continuing the installation process.
Fugaku is being developed not only for the progress in science, but also to help build the society dubbed as the “Society 5.0” by the Japanese government, where all people will live safe and comfortable lives. The current initiative to fight against the novel coronavirus is driven by the philosophy behind the development of Fugaku."
Initial Projects
Exploring new drug candidates for COVID-19 by "Fugaku"
Yasushi Okuno, RIKEN / Kyoto University
Prediction of conformational dynamics of proteins on the surface of SARS-Cov-2 using Fugaku
Yuji Sugita, RIKEN
Simulation analysis of pandemic phenomena
Nobuyasu Ito, RIKEN
Fragment molecular orbital calculations for COVID-19 proteins
Yuji Mochizuki, Rikkyo University
In this deck from the Performance Optimisation and Productivity group, Lubomir Riha from IT4Innovations presents: Energy Efficient Computing using Dynamic Tuning.
"We now live in a world of power-constrained architectures and systems and power consumption represents a significant cost factor in the overall HPC system economy. For these reasons, in recent years researchers, supercomputing centers and major vendors have developed new tools and methodologies to measure and optimize the energy consumption of large-scale high performance system installations. Due to the link between energy consumption, power consumption and execution time of an application executed by the final user, it is important for these tools and the methodology used to consider all these aspects, empowering the final user and the system administrator with the capability of finding the best configuration given different high level objectives.
This webinar focused on tools designed to improve the energy-efficiency of HPC applications using a methodology of dynamic tuning of HPC applications, developed under the H2020 READEX project. The READEX methodology has been designed for exploiting the dynamic behaviour of software. At design time, different runtime situations (RTS) are detected and optimized system configurations are determined. RTSs with the same configuration are grouped into scenarios, forming the tuning model. At runtime, the tuning model is used to switch system configurations dynamically.
The MERIC tool, that implements the READEX methodology, is presented. It supports manual or binary instrumentation of the analysed applications to simplify the analysis. This instrumentation is used to identify and annotate the significant regions in the HPC application. Automatic binary instrumentation annotates regions with significant runtime. Manual instrumentation, which can be combined with automatic, allows code developer to annotate regions of particular interest."
Watch the video: https://wp.me/p3RLHQ-lJP
Learn more: https://pop-coe.eu/blog/14th-pop-webinar-energy-efficient-computing-using-dynamic-tuning
and
https://code.it4i.cz/vys0053/meric
Sign up for our insideHPC Newsletter: http://insidehpc.com/newslett
In this deck from GTC Digital, William Beaudin from DDN presents: HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD.
Enabling high performance computing through the use of GPUs requires an incredible amount of IO to sustain application performance. We'll cover architectures that enable extremely scalable applications through the use of NVIDIA’s SuperPOD and DDN’s A3I systems.
The NVIDIA DGX SuperPOD is a first-of-its-kind artificial intelligence (AI) supercomputing infrastructure. DDN A³I with the EXA5 parallel file system is a turnkey, AI data storage infrastructure for rapid deployment, featuring faster performance, effortless scale, and simplified operations through deeper integration. The combined solution delivers groundbreaking performance, deploys in weeks as a fully integrated system, and is designed to solve the world's most challenging AI problems.
Watch the video: https://wp.me/p3RLHQ-lIV
Learn more: https://www.ddn.com/download/nvidia-superpod-ddn-a3i-ai400-appliance-with-the-exa5-filesystem/
and
https://www.nvidia.com/en-us/gtc/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck, Paul Isaacs from Linaro presents: State of ARM-based HPC. This talk provides an overview of applications and infrastructure services successfully ported to Aarch64 and benefiting from scale.
"With its debut on the TOP500, the 125,000-core Astra supercomputer at New Mexico’s Sandia Labs uses Cavium ThunderX2 chips to mark Arm’s entry into the petascale world. In Japan, the Fujitsu A64FX Arm-based CPU in the pending Fugaku supercomputer has been optimized to achieve high-level, real-world application performance, anticipating up to one hundred times the application execution performance of the K computer. K was the first computer to top 10 petaflops in 2011."
Watch the video: https://wp.me/p3RLHQ-lIT
Learn more: https://www.linaro.org/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Versal Premium ACAP for Network and Cloud Accelerationinside-BigData.com
Today Xilinx announced Versal Premium, the third series in the Versal ACAP portfolio. The Versal Premium series features highly integrated, networked and power-optimized cores and the industry’s highest bandwidth and compute density on an adaptable platform. Versal Premium is designed for the highest bandwidth networks operating in thermally and spatially constrained environments, as well as for cloud providers who need scalable, adaptable application acceleration.
Versal is the industry’s first adaptive compute acceleration platform (ACAP), a revolutionary new category of heterogeneous compute devices with capabilities that far exceed those of conventional silicon architectures. Developed on TSMC’s 7-nanometer process technology, Versal Premium combines software programmability with dynamically configurable hardware acceleration and pre-engineered connectivity and security features to enable a faster time-to- market. The Versal Premium series delivers up to 3X higher throughput compared to current generation FPGAs, with built-in Ethernet, Interlaken, and cryptographic engines that enable fast and secure networks. The series doubles the compute density of currently deployed mainstream FPGAs and provides the adaptability to keep pace with increasingly diverse and evolving cloud and networking workloads.
Learn more: https://insidehpc.com/2020/03/xilinx-announces-versal-premium-acap-for-network-and-cloud-acceleration/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Zettar: Moving Massive Amounts of Data across Any Distance Efficientlyinside-BigData.com
In this video from the Rice Oil & Gas Conference, Chin Fang from Zettar presents: Moving Massive Amounts of Data across Any Distance Efficiently.
The objective of this talk is to present two on-going projects aiming at improving and ensuring highly efficient bulk transferring or streaming of massive amounts of data over digital connections across any distance. It examines the current state of the art, a few very common misconceptions, the differences among the three major type of data movement solutions, a current initiative attempting to improve the data movement efficiency from the ground up, and another multi-stage project that shows how to conduct long distance large scale data movement at speed and scale internationally. Both projects have real world motivations, e.g. the ambitious data transfer requirements of Linac Coherent Light Source II (LCLS-II) [1], a premier preparation project of the U.S. DOE Exascale Computing Initiative (ECI) [2]. Their immediate goals are described and explained, together with the solution used for each. Findings and early results are reported. Possible future works are outlined.
Watch the video: https://wp.me/p3RLHQ-lBX
Learn more: https://www.zettar.com/
and
https://rice2020oghpc.rice.edu/program-2/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from the Rice Oil & Gas Conference, Bradley McCredie from AMD presents: Scaling TCO in a Post Moore's Law Era.
"While foundries bravely drive forward to overcome the technical and economic challenges posed by scaling to 5nm and beyond, Moore’s law alone can provide only a fraction of the performance / watt and performance / dollar gains needed to satisfy the demands of today’s high performance computing and artificial intelligence applications. To close the gap, multiple strategies are required. First, new levels of innovation and design efficiency will supplement technology gains to continue to deliver meaningful improvements in SoC performance. Second, heterogenous compute architectures will create x-factor increases of performance efficiency for the most critical applications. Finally, open software frameworks, APIs, and toolsets will enable broad ecosystems of application level innovation."
Watch the video:
Learn more: http://amd.com
and
https://rice2020oghpc.rice.edu/program-2/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
CUDA-Python and RAPIDS for blazing fast scientific computinginside-BigData.com
In this deck from the ECSS Symposium, Abe Stern from NVIDIA presents: CUDA-Python and RAPIDS for blazing fast scientific computing.
"We will introduce Numba and RAPIDS for GPU programming in Python. Numba allows us to write just-in-time compiled CUDA code in Python, giving us easy access to the power of GPUs from a powerful high-level language. RAPIDS is a suite of tools with a Python interface for machine learning and dataframe operations. Together, Numba and RAPIDS represent a potent set of tools for rapid prototyping, development, and analysis for scientific computing. We will cover the basics of each library and go over simple examples to get users started. Finally, we will briefly highlight several other relevant libraries for GPU programming."
Watch the video: https://wp.me/p3RLHQ-lvu
Learn more: https://developer.nvidia.com/rapids
and
https://www.xsede.org/for-users/ecss/ecss-symposium
Sign up for our insideHPC Newsletter: http://insidehp.com/newsletter
In this deck from FOSDEM 2020, Colin Sauze from Aberystwyth University describes the development of a RaspberryPi cluster for teaching an introduction to HPC.
"The motivation for this was to overcome four key problems faced by new HPC users:
* The availability of a real HPC system and the effect running training courses can have on the real system, conversely the availability of spare resources on the real system can cause problems for the training course.
* A fear of using a large and expensive HPC system for the first time and worries that doing something wrong might damage the system.
* That HPC systems are very abstract systems sitting in data centres that users never see, it is difficult for them to understand exactly what it is they are using.
* That new users fail to understand resource limitations, in part because of the vast resources in modern HPC systems a lot of mistakes can be made before running out of resources. A more resource constrained system makes it easier to understand this.
The talk will also discuss some of the technical challenges in deploying an HPC environment to a Raspberry Pi and attempts to keep that environment as close to a "real" HPC as possible. The issue to trying to automate the installation process will also be covered."
Learn more: https://github.com/colinsauze/pi_cluster
and
https://fosdem.org/2020/schedule/events/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from ATPESC 2019, Ken Raffenetti from Argonne presents an overview of HPC interconnects.
"The Argonne Training Program on Extreme-Scale Computing (ATPESC) provides intensive, two-week training on the key skills, approaches, and tools to design, implement, and execute computational science and engineering applications on current high-end computing systems and the leadership-class computing systems of the future."
Watch the video: https://wp.me/p3RLHQ-luc
Learn more: https://extremecomputingtraining.anl.gov/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Here’s a general definition of Big Data using the schema of the “four V’s” that’s become familiar. This isn’t specific to high performance data analysis. It applies to Big Data across all markets.To qualify as Big Data in this general context, the data set has to be large in volume, critical to analyze in a timeframe...It has to include multiple types of data and it has to be worthwhile to someone, preferably with a monetary value.
The emerging market for high performance data analysis is narrower than that. As I said a minute ago, it’s the market being formed by the convergence of data-intensive simulation and data-intensive analytical methods, so it’s really a union set. As the slide shows, this evolving market is very inclusive in relation to methods, types of data, and market sectors. The common denominator across these segments is the use of models that incorporate algorithmic complexity. You typically don’t find that kind of algorithmic complexity in online transaction processing or in commercial applications such as supply chain management and customer relationship management.The ultimate criterion for HPDA that it requires HPC resources.
There are important HPDA market drivers on the data ingestion side and the data output side. Data sources have become much more powerful. CERN’s Large Hadron Collider generates 1PB/second when it’s running. The Square Kilometer Array telescope will produce 1EB/day when it becomes operational in 2016. - But those are extreme examples. Much more common are sensor networks for power grids and other things, gene sequencers, MRI machines, and so on.- Onllne sales transactions produce a lot of data and a lot of opportunity for fraud. Standards, regulations and lawsuits are on the rise. Boeing stores all its engineering data for the 30-year lifetime of their commercial airplanes, not just as a reference for designing future planes but in case there’s a crash and a lawsuit. On the output side, more powerful HPC systems are kicking out lots more data in response to the growing user requirements you see listed here.
Moving data costs time and money. Energy has become very expensive. It can take 100 times more energy to move the results of a calculation than to perform the calculation in the first place. It’s no wonder that oil and gas companies, for example, still rely heavily on courier services for overnight shipping of disk drives. It would take too long and cost too much to send the data over a computer network.- If you’re a vendor, you have two main strategies available to you: you can speed up data movement , mainly through better interconnects, or you can minimize data movement by pre-filtering data or bringing the compute to the data, or you can both accelerate and minimize.
The data in most HPDA jobs assigned to HPC resources will continue to have regular access patterns, whether the data is structured or unstructured.This means it can be partitioned and mapped onto a standard cluster or other distributed memory machine for running Hadoop or other software.But there’s a rising tide of data work that exhibits irregular access patterns and can’t take advantage of data locality processing features. Caches are highly inefficient for jobs like this. These jobs benefit from global memory combined with powerful interconnects and other data movement capabilities. Partitionable jobs are very important now and non-partitionable jobs are becoming more important. By the way, SGI systems address both types. One general remark is that as the data analysis side of HPC expands, HPC architectures will need to become less compute-centric and offer more support for data integration and analysis.“Many current approaches to big data have been about ‘search’ – the ability to efficiently find something that you know is there in your data,” said Arvind Parthasarathi, President of YarcData. “uRiKA was purposely built to solve the problem of ‘discovery’ in big data – to discover things, relationships or patterns that you don’t know exist. By giving organizations the ability to do much faster hypothesis validation at scale and in real time, we are enabling the solution of business problems that were previously difficult or impossible – whether it be discovering the ideal patient treatment, investigating fraud, detecting threats, finding new trading algorithms or identifying counter-party risk. Basically, we are systematizing serendipity.”
HPC servers are often used for more than one purpose. IDC classifies HPC servers according to the primary purpose they’re used for. So, an HPDA server is one that’s used more than 50% for HPDA work. As this table shows, IDC forecasts that revenue for HPC servers acquired primarily for HPDA use will grow robustly (10.4% CAGR) to approach $1 billion in 2015. Because HPDA revenue starts as such a relatively small chunk of overall HPC server revenue, the HPDA share of the overall HPC server revenue will still be in the single digits in 2015, despite the fast growth rate.
Let’s look at some real-world use cases
This slide lists some of the most prominent use cases, meaning ones where repeated sales of HPC products have been happening. Fraud detection and life sciences are emerging fastest. BTW, I didn’t include financial services here because we’ve been tracking back-office FSI analytics as part of the HPC market for more than 20 years. But FSI is an important part of the high performance data analysis market. – not an easy one to penetrate for the first time.
I want to zero in more on the PayPal example because they gave me permission to use these slides and because in many ways they are representative of a larger group of commercial companies whose business requirements are pushing them up into HPC. The slides are from a talk PayPal gave IDC’s September 2012 HPC User Forum meeting in Dearborn, Michigan. By the way, if you want a copy of this talk or any of the long list of talks on one of our first slides, just email me at sconway [at] idc.com
PayPal is an eBay subsidiary and, among other things, has responsibility for detecting fraud across eBay and SKYPE. Five years ago, a day's worth of data was processed in batch processing overnight and fraud wasn't detected until as much as two weeks later. They realized they needed to detect fraud in real time, and for that they needed graph analysis. They were most interested in checking out collusion between multiple parties, such as when a credit card shows activity from four or more users. They needed to be able to stop that before the credit card got hit. IBM Watson on the Jeopardy game show was amazing but it was a needle in a haystack problem, meaning that Watson could only find answers that were already in its database. PayPal’s problem was different, because there was no visible needle to be found. Graph analysis let them uncover hidden relationships and behavior patterns
This gives you an idea of PayPal’s data volumes and HPDA requirements. These are going up all the time.
Here’s what PayPal is using. For the serious fraud detection and analysis, they’re using SGI servers and storage on an InfiniBand network. For the less-challenging work that doesn’t involve pattern discovery and real-time requirements, they’re running Hadoop on a cluster. By the way, PayPal says HPC has already saved them $710 million in fraud they wouldn’t have been able to detect before.
This gives you an idea of PayPal’s data volumes and HPDA requirements. These are going up all the time.
For cost and growth reasons, GEICO moved to automated insurance quotes on the phone. They needed to provide quotes instantaneously, in 100 milliseconds or less. They couldn’t do these calculations nearly fast enough on the fly .GEICO’s solution was to install an HPC system and every weekend run updated quotes for every adult and every household in the United States. That takes 60 wall clock hours today. The phones tap into the stored quotes and return the correct one in 100 milliseconds.
Here’s a real-world example of one of the biggest names in global package delivery. Their problem is not so different from PayPal’s. This courier service is doing real-time fraud detection on huge volumes of packages that come into their sorting facility from many locations and leave the facility for many other locations around the world.They ran a difficult benchmark. The winner hasn’t been publicly announced yet, but IDC’s back channels tell us the vendor has a 3-letter name that starts with S.
Schrödinger is a global life sciences software company with offices in Munich and Mannheim. One of the major things they do is use molecular dynamics to identify promising candidates for new drugs to combat cancer and other diseases – and it seems they’ve been using the cloud for this High Performance Data Analysis problem. That’s not so surprising, since molecular dynamics codes are often highly parallel.
Here’s the architecture they used. Note that they were already using HPC in their on premise data center, but the resources weren’t big enough for this task. That’s why they bursted out to Amazon EC2 using a software management layer from Cycle Computing to access more than 50,000 additional cores. Bringing a new drug to market can cost as much as £10 billion and a decade of time, so security is a major concern with commercial drug discovery. Apparently, Schrödinger felt confident about the cloud security measures.
You may have seen the recent news that Optum, which is part of United Health Group, is teaming with the Mayo Cline to build a huge center in Cambridge, Massachusetts to lay the research groundwork for outcomes-based medicine. They’ll have more than 100 million patient records at their disposal for this enormous data-intensive work.They’ll be using data-intensive methods to look at other aspects of health care, too. A week ago, United Health issued a press release in which they said they believe that improved efficiencies alone could reduce Medicare costs by about 40%, obviating much of the need for the major reforms the political parties have been fighting about.
In the U.S., the largest urban gangs are the Crips and the Bloods. They’re rival gangs that are at each other’s throats all the time, fighting for money and power. Both gangs are national in scope, but the national organizations aren’t that strong. The branches of these gangs in each city have a lot of autonomy to do what they want.What you see here, again in blurred form, was something that astounded the police department of Atlanta, Georgia, a city with about 4 million inhabitants. Through real-time monitory of social networks, they were able to witness, as it happened, the planned merger of these rival gangs in their city. This information allowed the police to adapt their own plans accordingly.
In summary, we defined HPDA and told you that IDC is forecasting rapid growth from a small base.HPDA is about the convergence of data-intensive HPC and high-end commercial analytics. One of the most interesting aspects of this, to us, is that the demands of the commercial market are moving this along faster in the commercial sector than in the traditional HPC market. PayPal is a great example of this (story of how PayPal was shy about presenting at User Forum – both sides should be learning from each other). On the analytics side, some attractive use cases are already out there. In the time allotted to us here, we described some of the more prominent ones, but there are many others.Most of the work will be done on clusters, but some economically important use cases need more capable architectures, especially for graph analytics.Many of the large investment firms are IDC clients, so our growth estimates tend to err on the side of conservatism. There is potential for the HPDA market to grow faster than our current forecast. But we talk with a lot of people and we update the forecasts often, so we get too far off the mark.
This is a partial list of the user and vendor talks on this topic that we’ve lined up in the past two years as part of the HPC User Forum. IDC has operated the HPC User Forum since 1999 for a volunteer steering committee made up of senior HPC people from government, industry and academia – organizations like Boeing, GM, Ford, NSF and others. We hold meetings across the world, and the talks listed here include perspectives on High Performance Data Analysis from the Americas, Europe and Asia.I’ll ask Chirag to explain how we define High Performance Data Analysis. I’ll return later to walk you through some real-world use cases. Chirag...