"Running high-performance scientific and engineering applications is challenging no matter where you do it. Join IT executives from Hitachi Global Storage Technology, The Aerospace Corporation, Novartis, and Cycle Computing and learn how they have used the AWS cloud to deploy mission-critical HPC workloads.
Cycle Computing leads the session on how organizations of any scale can run HPC workloads on AWS. Hitachi Global Storage Technology discusses experiences using the cloud to create next-generation hard drives. The Aerospace Corporation provides perspectives on running MPI and other simulations, and offer insights into considerations like security while running rocket science on the cloud. Novartis Institutes for Biomedical Research talks about a scientific computing environment to do performance benchmark workloads and large HPC clusters, including a 30,000-core environment for research in the fight against cancer, using the Cancer Genome Atlas (TCGA)."
Big Data and High Performance Computing Solutions in the AWS CloudAmazon Web Services
Managing big data and running supercomputing jobs used to be for only well-funded research organizations and large corporations, but not any longer. AWS has democratized supercomputing and big data for the masses! AWS can provide you with the 64th fastest supercomputer in the world, on-demand and pay as you go. Hear from Ben Butler, Head of AWS Big Data Marketing, to learn how our customers are using big data and high performance computing to change the world. Not only is AWS technology available to everyone, but it is self-service and cheaper than ever before, featuring innovative technology and flexible pricing models – our AWS cloud computing platform has disrupted big data and HPC. Learn from customer successes, as Ben shares real-world case studies describing the specific big data and high performance computing challenges being solved on AWS. We will conclude with a discussion around the tutorials, public datasets, test drives, and our grants program - all of the tools needed to get you started quickly.
AWS Webcast - An Introduction to High Performance Computing on AWSAmazon Web Services
High Performance Computing (HPC) allows scientists and engineers to solve complex science, engineering, and business problems using applications that require high bandwidth, low latency networking, and very high compute capabilities. Learn how the AWS cloud can cost- effectively provide the scalable computing resources, storage services, and analytic tools that enable running various kinds of HPC workloads. Who should attend? Engineers, architects, product managers, data scientists, high performance computing specialists, and researchers from industry and academia, along with technically-minded business stakeholders looking to put data to work for their organization.
Drug discovery at 2x speed. Faster, more comprehensive testing approval processes. Identifying gene targets in massive sequencing data sets. These goals are ambitious yet attainable, but not without increasing the computational capabilities of today's researchers. While everyone agrees that simply deploying more infrastructure is not the answer, running that work in the cloud is not without challenges. In this talk we will discuss and illustrate elements of those workloads that Cycle Computing's customers have run on AWS, generating vastly better results than would have been attained on traditional infrastructure. We will cover some common problems they encountered, and how they resolved them using Amazon EC2, S3, Glacier, and Cycle's software.
Presenters: Dougal Ballantyne, Business Development, AWS; Rob Futrick, CTO, Cycle Computing
Those who out-compute can many times out-compete. The cloud gives you access to a massive amount of compute power when you need it. This talk will present an introduction to HPC in the cloud, including, the benefits of HPC in the cloud, how to get started, some tools to use, and how you can manage data. We will showcase several examples of HPC in the cloud by a number of public sector and commercial customers.
Created by: Dr. Jeff Layton, Principal, Solutions Architect
Visit http:aws.amazon.com/hpc for more information about HPC on AWS.
High Performance Computing (HPC) allows scientists and engineers to solve complex science, engineering, and business problems using applications that require high bandwidth, low latency networking, and very high compute capabilities. AWS allows you to increase the speed of research by running high performance computing in the cloud and to reduce costs by providing Cluster Compute or Cluster GPU servers on-demand without large capital investments. You have access to a full-bisection, high bandwidth network for tightly-coupled, IO-intensive workloads, which enables you to scale out across thousands of cores for throughput-oriented applications.
(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014Amazon Web Services
Leveraging big data and high performance computing (HPC) solutions enables your organization to make smarter and faster decisions that influence strategy, increase productivity, and ultimately grow your business. We kick off the Big Data and HPC track with the latest advancements in data analytics, databases, storage, and HPC at AWS. Hear customer success stories and discover how to put data to work in your own organization.
(CMP202) Engineering Simulation and Analysis in the CloudAmazon Web Services
"Building great products, ones that are aesthetically appealing as well as functionally sound, requires cutting-edge design and engineering. Given the high cost of physical testing prototypes, engineering organizations are turning to simulation and analysis using digital models, but compute requirements for these have traditionally required expensive on-premises infrastructure. But now, engineering organizations can use high-performance computing services from AWS and solutions from AWS technology partners to innovate at scale globally, with no up-front capital infrastructure investment.
In this session, AWS Partner Ansys shares how they help customers of all sizes design and engineer better products through digital simulation and analysis using HPC on AWS."
Big Data and High Performance Computing Solutions in the AWS CloudAmazon Web Services
Managing big data and running supercomputing jobs used to be for only well-funded research organizations and large corporations, but not any longer. AWS has democratized supercomputing and big data for the masses! AWS can provide you with the 64th fastest supercomputer in the world, on-demand and pay as you go. Hear from Ben Butler, Head of AWS Big Data Marketing, to learn how our customers are using big data and high performance computing to change the world. Not only is AWS technology available to everyone, but it is self-service and cheaper than ever before, featuring innovative technology and flexible pricing models – our AWS cloud computing platform has disrupted big data and HPC. Learn from customer successes, as Ben shares real-world case studies describing the specific big data and high performance computing challenges being solved on AWS. We will conclude with a discussion around the tutorials, public datasets, test drives, and our grants program - all of the tools needed to get you started quickly.
AWS Webcast - An Introduction to High Performance Computing on AWSAmazon Web Services
High Performance Computing (HPC) allows scientists and engineers to solve complex science, engineering, and business problems using applications that require high bandwidth, low latency networking, and very high compute capabilities. Learn how the AWS cloud can cost- effectively provide the scalable computing resources, storage services, and analytic tools that enable running various kinds of HPC workloads. Who should attend? Engineers, architects, product managers, data scientists, high performance computing specialists, and researchers from industry and academia, along with technically-minded business stakeholders looking to put data to work for their organization.
Drug discovery at 2x speed. Faster, more comprehensive testing approval processes. Identifying gene targets in massive sequencing data sets. These goals are ambitious yet attainable, but not without increasing the computational capabilities of today's researchers. While everyone agrees that simply deploying more infrastructure is not the answer, running that work in the cloud is not without challenges. In this talk we will discuss and illustrate elements of those workloads that Cycle Computing's customers have run on AWS, generating vastly better results than would have been attained on traditional infrastructure. We will cover some common problems they encountered, and how they resolved them using Amazon EC2, S3, Glacier, and Cycle's software.
Presenters: Dougal Ballantyne, Business Development, AWS; Rob Futrick, CTO, Cycle Computing
Those who out-compute can many times out-compete. The cloud gives you access to a massive amount of compute power when you need it. This talk will present an introduction to HPC in the cloud, including, the benefits of HPC in the cloud, how to get started, some tools to use, and how you can manage data. We will showcase several examples of HPC in the cloud by a number of public sector and commercial customers.
Created by: Dr. Jeff Layton, Principal, Solutions Architect
Visit http:aws.amazon.com/hpc for more information about HPC on AWS.
High Performance Computing (HPC) allows scientists and engineers to solve complex science, engineering, and business problems using applications that require high bandwidth, low latency networking, and very high compute capabilities. AWS allows you to increase the speed of research by running high performance computing in the cloud and to reduce costs by providing Cluster Compute or Cluster GPU servers on-demand without large capital investments. You have access to a full-bisection, high bandwidth network for tightly-coupled, IO-intensive workloads, which enables you to scale out across thousands of cores for throughput-oriented applications.
(BDT201) Big Data and HPC State of the Union | AWS re:Invent 2014Amazon Web Services
Leveraging big data and high performance computing (HPC) solutions enables your organization to make smarter and faster decisions that influence strategy, increase productivity, and ultimately grow your business. We kick off the Big Data and HPC track with the latest advancements in data analytics, databases, storage, and HPC at AWS. Hear customer success stories and discover how to put data to work in your own organization.
(CMP202) Engineering Simulation and Analysis in the CloudAmazon Web Services
"Building great products, ones that are aesthetically appealing as well as functionally sound, requires cutting-edge design and engineering. Given the high cost of physical testing prototypes, engineering organizations are turning to simulation and analysis using digital models, but compute requirements for these have traditionally required expensive on-premises infrastructure. But now, engineering organizations can use high-performance computing services from AWS and solutions from AWS technology partners to innovate at scale globally, with no up-front capital infrastructure investment.
In this session, AWS Partner Ansys shares how they help customers of all sizes design and engineer better products through digital simulation and analysis using HPC on AWS."
How to calculate the cost of a Hadoop infrastructure on Amazon AWS, given some data volume estimates and the rough use case ?
Presentation attempts to compare the different options available on AWS.
The TCO Calculator - Estimate the True Cost of Hadoop MapR Technologies
http://bit.ly/1wsAuRS - There are many hidden costs for Apache Hadoop that have different effects across different Hadoop distributions. With the new MapR TCO calculator organisations have a simple and reliable tool that is based on facts to compare costs.
Learn from Accubits Technologies
High Performance Computing (HPC) most generally refers to the practice of aggregating computing power in a way that delivers much higher performance than one could get out of a typical desktop computer or workstation in order to solve large problems in science, engineering, or business.
Auto Scaling Systems With Elastic Spark Streaming: Spark Summit East talk by ...Spark Summit
Come explore a feature we’ve created that is not supported out-of-the-box: the ability to add or remove nodes to always-on real time Spark Streaming jobs. Elastic Spark Streaming jobs can automatically adjust to the demands of traffic or volume. Using a set of configurable utility classes, these jobs scale down when lulls are detected and scale up when load is too high. We process multiple TB’s per day with billions of events. Our traffic pattern experiences natural peaks and valleys with the occasional sustained unexpected spike. Elastic jobs has freed us from manual intervention, given back developer time, and has made a large financial impact through maximized resource utilization.
In this sessions, learn the basics of HPC on AWS and watch a demonstration on how to launch a large cluster.
Presenter: Scott Eberhardt, Specialist Solutions Architect, AWS UK
Challenges for running Hadoop on AWS - AdvancedAWS MeetupAndrei Savu
Nowadays we've got all the tools we need to spin-up and tear-down clusters with hundreds of nodes in minutes and this puts more pressure on the tools we use to configure and monitor our applications. This challenge is even more interesting when we have to deal with long running distributed data storage and processing systems like Hadoop. In this talk we will look into some of the challenges we need to deal with when creating and managing Hadoop clusters in AWS, we will discuss improvement opportunities in monitoring (e.g. detecting and dealing with instance failure, resource contention & noisy neighbors) and a bit about the future and how we should go about disconnecting workload dispatch from cluster lifecycle.
The Pandemic Changes Everything, the Need for Speed and ResiliencyAlluxio, Inc.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
The Pandemic Changes Everything, the Need for Speed and Resiliency
Parviz Peiravi, Global CTO of Financial Services Solutions, Intel
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...Spark Summit
Redis accelerates Apache Spark execution by 45 times, when used as a shared distributed in-memory datastore for Spark in analyses like time series data range queries. With the redis module for machine learning, redis-ml, implementation of spark-ml models gains a new real time serving layer that offloads processing of models directly in Redis, allows multiple applications to reuse the same models and speeds up classification and execution of these models by 13x. Join this session to learn more about the Redis Labs’ connector for Apache Spark that enhances production implementations of real-time big data processing.
High Performance Computing on AWS: Accelerating Innovation with virtually unl...Amazon Web Services
In this session, learn how you innovate without limits, reduce costs, and get your results to market faster by moving your HPC workloads to AWS. Learn how you can use HPC on AWS to let your research needs dictate you HPC architecture requirements, not the other way around. Understand how to create, operate, and tear down secure, well-optimized HPC clusters in minutes.
Deciding the deployment model is critical when enterprises adopt Hadoop. Initially, the bare metal (on-premise cluster with physical servers) model was popular to avoid I/O overhead in the virtualized environments. However, these days, cloud is also a contending option with its compelling cost savings, and ease of operation. To aid in assessing the deployment options, Accenture Technology Labs developed Accenture Data Platform Benchmark suite, a total cost of ownership (TCO) model and has tuned and compared performance of bare metal Hadoop clusters and Hadoop cloud service. Interestingly enough, the study discovered that price/performance ratio is not a critical factor in making a Hadoop deployment decision. Employing empirical and systemic analyses, the study resulted in comparable price/performance ratio from both bare metal Hadoop clusters and Hadoop-as-a-service. Moreover, cheaper purchasing options (e.g., long term contracts) provides better ratio than the bare metal one in many cases. Thus, this result debunks the idea that the cloud is not suitable to Hadoop MapReduce workloads due to their heavy I/O requirements. Furthermore, the study finds that the Hadoop default configuration provides ample headroom for performance tuning, and the cloud infrastructure enables even further performance tuning opportunities.
High Performance Computing (HPC) has been driving technology advancements for many decades. HPC enables performance-demanding applications and workloads to solve complex problems while dramatically reducing time to solution. With a history of requiring very large data centers, HPC is now on the edge of a paradigm shift. The AWS Cloud will allow customers to have access to near infinite compute and storage resources, without the overhead of running their own data centers. There are a vast number of HPC segments and verticals that are already seeing great success running their workloads on AWS. Life Sciences, Financial Services, Energy & Geo Sciences, as well as Manufacturing are successfully deploying their applications on AWS. In these two sessions we will discuss how AWS can help you run HPC workloads in the cloud. The first session will be a general introduction to HPC on AWS.
Accelerating analytics in the cloud with the Starburst Presto + Alluxio stackAlluxio, Inc.
Alluxio Tech Talk
January 21, 2020
Speakers:
Matt Fuller, Starburst
Dipti Borkar, Alluxio
With the advent of the public clouds and data increasingly siloed across many locations -- on premises and in the public cloud -- enterprises are looking for more flexibility and higher performance approaches to analyze their structured data.
Join us for this tech talk where we’ll introduce the Starburst Presto, Alluxio, and cloud object store stack for building a highly-concurrent and low-latency analytics platform. This stack provides a strong solution to run fast SQL across multiple storage systems including HDFS, S3, and others in public cloud, hybrid cloud, and multi-cloud environments. You’ll learn more about:
- The architecture of Presto, an open source distributed SQL engine
- How the Presto + Alluxio stack queries data from cloud object storage like S3 for faster and more cost-effective analytics
- Achieving data locality and cross-job caching with Alluxio regardless of where data is persisted
Technical computing (high-performance computing) used to be the domain of specialists using expensive, proprietary equipment. Today, technical computing is going mainstream, becoming the absolutely irreplaceable competitive tool for research scientists and businesses alike.
Here's a look at Dell’s pioneering role in the evolution of technical computing, with a focus on the key industry trends and technologies that will bring the next generation of tools and functionality to research and development organizations around the world.
Building a Just-in-Time Application Stack for AnalystsAvere Systems
Slide presentation from Webinar on February 17, 2016.
People in analytical roles are demanding more and more compute and storage to get their jobs done. Instead of building out infrastructure for a few employees or a department, systems engineers and IT managers can find value in creating a compute stack in the cloud to meet the fluctuating demand of their clients.
In this 45-minute webinar, you’ll learn:
- How to identify the right analytical workloads
- How to create a scalable compute environment using the cloud for analysts in under 10 minutes
- How to best manage costs associated with the cloud compute stack
- How to create dedicated client stacks with their own scratch space as well as general access to reference data
Health systems departments, research & development departments, and business analyst groups all face silos of these challenging, compute-intensive use cases. By learning how to quickly build this flexible workflow that can be scaled up and down (or off) instantly, you can support business objectives while efficiently managing costs.
Ultra Fast Deep Learning in Hybrid Cloud Using Intel Analytics Zoo & AlluxioAlluxio, Inc.
Alluxio Global Online Meetup
Apr 23, 2020
For more Alluxio events: https://www.alluxio.io/events/
Speakers:
Jiao (Jennie) Wang, Intel
Tsai Louie, Intel
Bin Fan, Alluxio
Today, many people run deep learning applications with training data from separate storage such as object storage or remote data centers. This presentation will demo the Intel Analytics Zoo + Alluxio stack, an architecture that enables high performance while keeping cost and resource efficiency balanced without network being I/O bottlenecked.
Intel Analytics Zoo is a unified data analytics and AI platform open-sourced by Intel. It seamlessly unites TensorFlow, Keras, PyTorch, Spark, Flink, and Ray programs into an integrated pipeline, which can transparently scale from a laptop to large clusters to process production big data. Alluxio, as an open-source data orchestration layer, accelerates data loading and processing in Analytics Zoo deep learning applications.
This talk, we will go over:
- What is Analytics Zoo and how it works
- How to run Analytics Zoo with Alluxio in deep learning applications
- Initial performance benchmark results using the Analytics Zoo + Alluxio stack
Deep Learning in the Cloud at Scale: A Data Orchestration StoryAlluxio, Inc.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Deep Learning in the Cloud at Scale: A Data Orchestration Story
Mickey Zhang, Software Engineer (Microsoft)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
How to calculate the cost of a Hadoop infrastructure on Amazon AWS, given some data volume estimates and the rough use case ?
Presentation attempts to compare the different options available on AWS.
The TCO Calculator - Estimate the True Cost of Hadoop MapR Technologies
http://bit.ly/1wsAuRS - There are many hidden costs for Apache Hadoop that have different effects across different Hadoop distributions. With the new MapR TCO calculator organisations have a simple and reliable tool that is based on facts to compare costs.
Learn from Accubits Technologies
High Performance Computing (HPC) most generally refers to the practice of aggregating computing power in a way that delivers much higher performance than one could get out of a typical desktop computer or workstation in order to solve large problems in science, engineering, or business.
Auto Scaling Systems With Elastic Spark Streaming: Spark Summit East talk by ...Spark Summit
Come explore a feature we’ve created that is not supported out-of-the-box: the ability to add or remove nodes to always-on real time Spark Streaming jobs. Elastic Spark Streaming jobs can automatically adjust to the demands of traffic or volume. Using a set of configurable utility classes, these jobs scale down when lulls are detected and scale up when load is too high. We process multiple TB’s per day with billions of events. Our traffic pattern experiences natural peaks and valleys with the occasional sustained unexpected spike. Elastic jobs has freed us from manual intervention, given back developer time, and has made a large financial impact through maximized resource utilization.
In this sessions, learn the basics of HPC on AWS and watch a demonstration on how to launch a large cluster.
Presenter: Scott Eberhardt, Specialist Solutions Architect, AWS UK
Challenges for running Hadoop on AWS - AdvancedAWS MeetupAndrei Savu
Nowadays we've got all the tools we need to spin-up and tear-down clusters with hundreds of nodes in minutes and this puts more pressure on the tools we use to configure and monitor our applications. This challenge is even more interesting when we have to deal with long running distributed data storage and processing systems like Hadoop. In this talk we will look into some of the challenges we need to deal with when creating and managing Hadoop clusters in AWS, we will discuss improvement opportunities in monitoring (e.g. detecting and dealing with instance failure, resource contention & noisy neighbors) and a bit about the future and how we should go about disconnecting workload dispatch from cluster lifecycle.
The Pandemic Changes Everything, the Need for Speed and ResiliencyAlluxio, Inc.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
The Pandemic Changes Everything, the Need for Speed and Resiliency
Parviz Peiravi, Global CTO of Financial Services Solutions, Intel
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Analytics at the Real-Time Speed of Business: Spark Summit East talk by Manis...Spark Summit
Redis accelerates Apache Spark execution by 45 times, when used as a shared distributed in-memory datastore for Spark in analyses like time series data range queries. With the redis module for machine learning, redis-ml, implementation of spark-ml models gains a new real time serving layer that offloads processing of models directly in Redis, allows multiple applications to reuse the same models and speeds up classification and execution of these models by 13x. Join this session to learn more about the Redis Labs’ connector for Apache Spark that enhances production implementations of real-time big data processing.
High Performance Computing on AWS: Accelerating Innovation with virtually unl...Amazon Web Services
In this session, learn how you innovate without limits, reduce costs, and get your results to market faster by moving your HPC workloads to AWS. Learn how you can use HPC on AWS to let your research needs dictate you HPC architecture requirements, not the other way around. Understand how to create, operate, and tear down secure, well-optimized HPC clusters in minutes.
Deciding the deployment model is critical when enterprises adopt Hadoop. Initially, the bare metal (on-premise cluster with physical servers) model was popular to avoid I/O overhead in the virtualized environments. However, these days, cloud is also a contending option with its compelling cost savings, and ease of operation. To aid in assessing the deployment options, Accenture Technology Labs developed Accenture Data Platform Benchmark suite, a total cost of ownership (TCO) model and has tuned and compared performance of bare metal Hadoop clusters and Hadoop cloud service. Interestingly enough, the study discovered that price/performance ratio is not a critical factor in making a Hadoop deployment decision. Employing empirical and systemic analyses, the study resulted in comparable price/performance ratio from both bare metal Hadoop clusters and Hadoop-as-a-service. Moreover, cheaper purchasing options (e.g., long term contracts) provides better ratio than the bare metal one in many cases. Thus, this result debunks the idea that the cloud is not suitable to Hadoop MapReduce workloads due to their heavy I/O requirements. Furthermore, the study finds that the Hadoop default configuration provides ample headroom for performance tuning, and the cloud infrastructure enables even further performance tuning opportunities.
High Performance Computing (HPC) has been driving technology advancements for many decades. HPC enables performance-demanding applications and workloads to solve complex problems while dramatically reducing time to solution. With a history of requiring very large data centers, HPC is now on the edge of a paradigm shift. The AWS Cloud will allow customers to have access to near infinite compute and storage resources, without the overhead of running their own data centers. There are a vast number of HPC segments and verticals that are already seeing great success running their workloads on AWS. Life Sciences, Financial Services, Energy & Geo Sciences, as well as Manufacturing are successfully deploying their applications on AWS. In these two sessions we will discuss how AWS can help you run HPC workloads in the cloud. The first session will be a general introduction to HPC on AWS.
Accelerating analytics in the cloud with the Starburst Presto + Alluxio stackAlluxio, Inc.
Alluxio Tech Talk
January 21, 2020
Speakers:
Matt Fuller, Starburst
Dipti Borkar, Alluxio
With the advent of the public clouds and data increasingly siloed across many locations -- on premises and in the public cloud -- enterprises are looking for more flexibility and higher performance approaches to analyze their structured data.
Join us for this tech talk where we’ll introduce the Starburst Presto, Alluxio, and cloud object store stack for building a highly-concurrent and low-latency analytics platform. This stack provides a strong solution to run fast SQL across multiple storage systems including HDFS, S3, and others in public cloud, hybrid cloud, and multi-cloud environments. You’ll learn more about:
- The architecture of Presto, an open source distributed SQL engine
- How the Presto + Alluxio stack queries data from cloud object storage like S3 for faster and more cost-effective analytics
- Achieving data locality and cross-job caching with Alluxio regardless of where data is persisted
Technical computing (high-performance computing) used to be the domain of specialists using expensive, proprietary equipment. Today, technical computing is going mainstream, becoming the absolutely irreplaceable competitive tool for research scientists and businesses alike.
Here's a look at Dell’s pioneering role in the evolution of technical computing, with a focus on the key industry trends and technologies that will bring the next generation of tools and functionality to research and development organizations around the world.
Building a Just-in-Time Application Stack for AnalystsAvere Systems
Slide presentation from Webinar on February 17, 2016.
People in analytical roles are demanding more and more compute and storage to get their jobs done. Instead of building out infrastructure for a few employees or a department, systems engineers and IT managers can find value in creating a compute stack in the cloud to meet the fluctuating demand of their clients.
In this 45-minute webinar, you’ll learn:
- How to identify the right analytical workloads
- How to create a scalable compute environment using the cloud for analysts in under 10 minutes
- How to best manage costs associated with the cloud compute stack
- How to create dedicated client stacks with their own scratch space as well as general access to reference data
Health systems departments, research & development departments, and business analyst groups all face silos of these challenging, compute-intensive use cases. By learning how to quickly build this flexible workflow that can be scaled up and down (or off) instantly, you can support business objectives while efficiently managing costs.
Ultra Fast Deep Learning in Hybrid Cloud Using Intel Analytics Zoo & AlluxioAlluxio, Inc.
Alluxio Global Online Meetup
Apr 23, 2020
For more Alluxio events: https://www.alluxio.io/events/
Speakers:
Jiao (Jennie) Wang, Intel
Tsai Louie, Intel
Bin Fan, Alluxio
Today, many people run deep learning applications with training data from separate storage such as object storage or remote data centers. This presentation will demo the Intel Analytics Zoo + Alluxio stack, an architecture that enables high performance while keeping cost and resource efficiency balanced without network being I/O bottlenecked.
Intel Analytics Zoo is a unified data analytics and AI platform open-sourced by Intel. It seamlessly unites TensorFlow, Keras, PyTorch, Spark, Flink, and Ray programs into an integrated pipeline, which can transparently scale from a laptop to large clusters to process production big data. Alluxio, as an open-source data orchestration layer, accelerates data loading and processing in Analytics Zoo deep learning applications.
This talk, we will go over:
- What is Analytics Zoo and how it works
- How to run Analytics Zoo with Alluxio in deep learning applications
- Initial performance benchmark results using the Analytics Zoo + Alluxio stack
Deep Learning in the Cloud at Scale: A Data Orchestration StoryAlluxio, Inc.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Deep Learning in the Cloud at Scale: A Data Orchestration Story
Mickey Zhang, Software Engineer (Microsoft)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Solving enterprise challenges through scale out storage & big compute finalAvere Systems
Google Cloud Platform, Avere Systems, and Cycle Computing experts will share best practices for advancing solutions to big challenges faced by enterprises with growing compute and storage needs. In this “best practices” webinar, you’ll hear how these companies are working to improve results that drive businesses forward through scalability, performance, and ease of management.
The slides were from a webinar presented January 24, 2017. The audience learned:
- How enterprises are using Google Cloud Platform to gain compute and storage capacity on-demand
- Best practices for efficient use of cloud compute and storage resources
- Overcoming the need for file systems within a hybrid cloud environment
- Understand how to eliminate latency between cloud and data center architectures
- Learn how to best manage simulation, analytics, and big data workloads in dynamic environments
- Look at market dynamics drawing companies to new storage models over the next several years
Presenters communicated a foundation to build infrastructure to support ongoing demand growth.
Accelerate Analytics and ML in the Hybrid Cloud EraAlluxio, Inc.
Alluxio Webinar
April 6, 2021
For more Alluxio events: https://www.alluxio.io/events/
Speakers:
Alex Ma, Alluxio
Peter Behrakis, Alluxio
Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows.
In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see.
In this tech talk, we'll go over:
- What is Alluxio Data Orchestration?
- How does it work?
- Alluxio customer results
ICP for Data- Enterprise platform for AI, ML and Data ScienceKaran Sachdeva
IBM Cloud Private for Data, an ultimate platform for all AI, ML and Data Science workloads. Integrated analytics platform based on Containers and micro services. Works with Kubernetes and dockers, even with Redhat openshift. Delivers the variety of business use cases in all industries- FS, Telco, Retail, Manufacturing etc
ADV Slides: Comparing the Enterprise Analytic SolutionsDATAVERSITY
Data is the foundation of any meaningful corporate initiative. Fully master the necessary data, and you’re more than halfway to success. That’s why leverageable (i.e., multiple use) artifacts of the enterprise data environment are so critical to enterprise success.
Build them once (keep them updated), and use again many, many times for many and diverse ends. The data warehouse remains focused strongly on this goal. And that may be why, nearly 40 years after the first database was labeled a “data warehouse,” analytic database products still target the data warehouse.
ADV Slides: When and How Data Lakes Fit into a Modern Data ArchitectureDATAVERSITY
Whether to take data ingestion cycles off the ETL tool and the data warehouse or to facilitate competitive Data Science and building algorithms in the organization, the data lake – a place for unmodeled and vast data – will be provisioned widely in 2020.
Though it doesn’t have to be complicated, the data lake has a few key design points that are critical, and it does need to follow some principles for success. Avoid building the data swamp, but not the data lake! The tool ecosystem is building up around the data lake and soon many will have a robust lake and data warehouse. We will discuss policy to keep them straight, send data to its best platform, and keep users’ confidence up in their data platforms.
Data lakes will be built in cloud object storage. We’ll discuss the options there as well.
Get this data point for your data lake journey.
Imagine an entire IT infrastructure controlled not by hands and hardware, but by software. One in which application workloads such as big data, analytics, simulation and design are serviced automatically by the most appropriate resource, whether running locally or in the cloud. A Software Defined Infrastructure enables your organization to deliver IT services in the most efficient way possible, optimizing resource utilization to accelerate time to results and reduce costs. It is the foundation for a fully integrated software defined environment, optimizing your compute, storage and networking infrastructure so you can quickly adapt to changing business requirements. A comprehensive portfolio of management tools dynamically manage workloads and data, transforming a static IT infrastructure into a workload- , resource- and data-aware environment.
Learn more: http://ibm.co/1wkoXtc
Watch the video presentation: http://insidehpc.com/2015/03/slidecast-software-defined-infrastructure/
Big Data, IoT, data lake, unstructured data, Hadoop, cloud, and massively parallel processing (MPP) are all just fancy words unless you can find uses cases for all this technology. Join me as I talk about the many use cases I have seen, from streaming data to advanced analytics, broken down by industry. I’ll show you how all this technology fits together by discussing various architectures and the most common approaches to solving data problems and hopefully set off light bulbs in your head on how big data can help your organization make better business decisions.
Design Choices for Cloud Data PlatformsAshish Mrig
You have decided to migrate your workload to Cloud, congratulations ! Which database should be used to host and query your data ? Most people go default: AWS -> Redshift, GCP ->BigQuery, Azure -> Synapse and so on. This presentation will go over design considerations, guidelines and best practices to choose your data platform and will go beyond the default choices. We will talk about evolutions of databases, design, data modeling and how to minimize the cost.
Maximizing Oil and Gas (Data) Asset Utilization with a Logical Data Fabric (A...Denodo
Watch full webinar here: https://bit.ly/3g9PlQP
It is no news that Oil and Gas companies are constantly faced with immense pressure to stay competitive, especially in the current climate while striving towards becoming data-driven at the heart of the process to scale and gain greater operational efficiencies across the organization.
Hence, the need for a logical data layer to help Oil and Gas businesses move towards a unified secure and governed environment to optimize the potential of data assets across the enterprise efficiently and deliver real-time insights.
Tune in to this on-demand webinar where you will:
- Discover the role of data fabrics and Industry 4.0 in enabling smart fields
- Understand how to connect data assets and the associated value chain to high impact domain areas
- See examples of organizations accelerating time-to-value and reducing NPT
- Learn best practices for handling real-time/streaming/IoT data for analytical and operational use cases
GCP On Prem Buyers Guide - White-paper | Qubole Vasu S
A buyer's guide for migrating a data lake to google cloud, we look at the efficiency and agility an organization can achieve by adopting the qubole open data lake platform & google cloud platform
https://www.qubole.com/resources/white-papers/gcp-on-prem-buyers-guide
Bitkom Cray presentation - on HPC affecting big data analytics in FSPhilip Filleul
High value analytics in FS are being enabled by Graph, machine learning and Spark technologies. To make these real at production scale HPC technologies are more appropriate than commodity clusters.
Similar to Real-world Cloud HPC at Scale, for Production Workloads (BDT212) | AWS re:Invent 2013 (20)
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
Il Forecasting è un processo importante per tantissime aziende e viene utilizzato in vari ambiti per cercare di prevedere in modo accurato la crescita e distribuzione di un prodotto, l’utilizzo delle risorse necessarie nelle linee produttive, presentazioni finanziarie e tanto altro. Amazon utilizza delle tecniche avanzate di forecasting, in parte questi servizi sono stati messi a disposizione di tutti i clienti AWS.
In questa sessione illustreremo come pre-processare i dati che contengono una componente temporale e successivamente utilizzare un algoritmo che a partire dal tipo di dato analizzato produce un forecasting accurato.
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
La varietà e la quantità di dati che si crea ogni giorno accelera sempre più velocemente e rappresenta una opportunità irripetibile per innovare e creare nuove startup.
Tuttavia gestire grandi quantità di dati può apparire complesso: creare cluster Big Data su larga scala sembra essere un investimento accessibile solo ad aziende consolidate. Ma l’elasticità del Cloud e, in particolare, i servizi Serverless ci permettono di rompere questi limiti.
Vediamo quindi come è possibile sviluppare applicazioni Big Data rapidamente, senza preoccuparci dell’infrastruttura, ma dedicando tutte le risorse allo sviluppo delle nostre le nostre idee per creare prodotti innovativi.
Ora puoi utilizzare Amazon Elastic Kubernetes Service (EKS) per eseguire pod Kubernetes su AWS Fargate, il motore di elaborazione serverless creato per container su AWS. Questo rende più semplice che mai costruire ed eseguire le tue applicazioni Kubernetes nel cloud AWS.In questa sessione presenteremo le caratteristiche principali del servizio e come distribuire la tua applicazione in pochi passaggi
Vent'anni fa Amazon ha attraversato una trasformazione radicale con l'obiettivo di aumentare il ritmo dell'innovazione. In questo periodo abbiamo imparato come cambiare il nostro approccio allo sviluppo delle applicazioni ci ha permesso di aumentare notevolmente l'agilità, la velocità di rilascio e, in definitiva, ci ha consentito di creare applicazioni più affidabili e scalabili. In questa sessione illustreremo come definiamo le applicazioni moderne e come la creazione di app moderne influisce non solo sull'architettura dell'applicazione, ma sulla struttura organizzativa, sulle pipeline di rilascio dello sviluppo e persino sul modello operativo. Descriveremo anche approcci comuni alla modernizzazione, compreso l'approccio utilizzato dalla stessa Amazon.com.
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
L’utilizzo dei container è in continua crescita.
Se correttamente disegnate, le applicazioni basate su Container sono molto spesso stateless e flessibili.
I servizi AWS ECS, EKS e Kubernetes su EC2 possono sfruttare le istanze Spot, portando ad un risparmio medio del 70% rispetto alle istanze On Demand. In questa sessione scopriremo insieme quali sono le caratteristiche delle istanze Spot e come possono essere utilizzate facilmente su AWS. Impareremo inoltre come Spreaker sfrutta le istanze spot per eseguire applicazioni di diverso tipo, in produzione, ad una frazione del costo on-demand!
In recent months, many customers have been asking us the question – how to monetise Open APIs, simplify Fintech integrations and accelerate adoption of various Open Banking business models. Therefore, AWS and FinConecta would like to invite you to Open Finance marketplace presentation on October 20th.
Event Agenda :
Open banking so far (short recap)
• PSD2, OB UK, OB Australia, OB LATAM, OB Israel
Intro to Open Finance marketplace
• Scope
• Features
• Tech overview and Demo
The role of the Cloud
The Future of APIs
• Complying with regulation
• Monetizing data / APIs
• Business models
• Time to market
One platform for all: a Strategic approach
Q&A
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
Per creare valore e costruire una propria offerta differenziante e riconoscibile, le startup di successo sanno come combinare tecnologie consolidate con componenti innovativi creati ad hoc.
AWS fornisce servizi pronti all'utilizzo e, allo stesso tempo, permette di personalizzare e creare gli elementi differenzianti della propria offerta.
Concentrandoci sulle tecnologie di Machine Learning, vedremo come selezionare i servizi di intelligenza artificiale offerti da AWS e, anche attraverso una demo, come costruire modelli di Machine Learning personalizzati utilizzando SageMaker Studio.
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
Con l'approccio tradizionale al mondo IT per molti anni è stato difficile implementare tecniche di DevOps, che finora spesso hanno previsto attività manuali portando di tanto in tanto a dei downtime degli applicativi interrompendo l'operatività dell'utente. Con l'avvento del cloud, le tecniche di DevOps sono ormai a portata di tutti a basso costo per qualsiasi genere di workload, garantendo maggiore affidabilità del sistema e risultando in dei significativi miglioramenti della business continuity.
AWS mette a disposizione AWS OpsWork come strumento di Configuration Management che mira ad automatizzare e semplificare la gestione e i deployment delle istanze EC2 per mezzo di workload Chef e Puppet.
Scopri come sfruttare AWS OpsWork a garanzia e affidabilità del tuo applicativo installato su Instanze EC2.
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
Vuoi conoscere le opzioni per eseguire Microsoft Active Directory su AWS? Quando si spostano carichi di lavoro Microsoft in AWS, è importante considerare come distribuire Microsoft Active Directory per supportare la gestione, l'autenticazione e l'autorizzazione dei criteri di gruppo. In questa sessione, discuteremo le opzioni per la distribuzione di Microsoft Active Directory su AWS, incluso AWS Directory Service per Microsoft Active Directory e la distribuzione di Active Directory su Windows su Amazon Elastic Compute Cloud (Amazon EC2). Trattiamo argomenti quali l'integrazione del tuo ambiente Microsoft Active Directory locale nel cloud e l'utilizzo di applicazioni SaaS, come Office 365, con AWS Single Sign-On.
Dal riconoscimento facciale al riconoscimento di frodi o difetti di fabbricazione, l'analisi di immagini e video che sfruttano tecniche di intelligenza artificiale, si stanno evolvendo e raffinando a ritmi elevati. In questo webinar esploreremo le possibilità messe a disposizione dai servizi AWS per applicare lo stato dell'arte delle tecniche di computer vision a scenari reali.
Amazon Web Services e VMware organizzano un evento virtuale gratuito il prossimo mercoledì 14 Ottobre dalle 12:00 alle 13:00 dedicato a VMware Cloud ™ on AWS, il servizio on demand che consente di eseguire applicazioni in ambienti cloud basati su VMware vSphere® e di accedere ad una vasta gamma di servizi AWS, sfruttando a pieno le potenzialità del cloud AWS e tutelando gli investimenti VMware esistenti.
Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi.
La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
Molte aziende oggi, costruiscono applicazioni con funzionalità di tipo ledger ad esempio per verificare lo storico di accrediti o addebiti nelle transazioni bancarie o ancora per tenere traccia del flusso supply chain dei propri prodotti.
Alla base di queste soluzioni ci sono i database ledger che permettono di avere un log delle transazioni trasparente, immutabile e crittograficamente verificabile, ma sono strumenti complessi e onerosi da gestire.
Amazon QLDB elimina la necessità di costruire sistemi personalizzati e complessi fornendo un database ledger serverless completamente gestito.
In questa sessione scopriremo come realizzare un'applicazione serverless completa che utilizzi le funzionalità di QLDB.
Con l’ascesa delle architetture di microservizi e delle ricche applicazioni mobili e Web, le API sono più importanti che mai per offrire agli utenti finali una user experience eccezionale. In questa sessione impareremo come affrontare le moderne sfide di progettazione delle API con GraphQL, un linguaggio di query API open source utilizzato da Facebook, Amazon e altro e come utilizzare AWS AppSync, un servizio GraphQL serverless gestito su AWS. Approfondiremo diversi scenari, comprendendo come AppSync può aiutare a risolvere questi casi d’uso creando API moderne con funzionalità di aggiornamento dati in tempo reale e offline.
Inoltre, impareremo come Sky Italia utilizza AWS AppSync per fornire aggiornamenti sportivi in tempo reale agli utenti del proprio portale web.
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
Molte organizzazioni sfruttano i vantaggi del cloud migrando i propri carichi di lavoro Oracle e assicurandosi notevoli vantaggi in termini di agilità ed efficienza dei costi.
La migrazione di questi carichi di lavoro, può creare complessità durante la modernizzazione e il refactoring delle applicazioni e a questo si possono aggiungere rischi di prestazione che possono essere introdotti quando si spostano le applicazioni dai data center locali.
In queste slide, gli esperti AWS e VMware presentano semplici e pratici accorgimenti per facilitare e semplificare la migrazione dei carichi di lavoro Oracle accelerando la trasformazione verso il cloud, approfondiranno l’architettura e dimostreranno come sfruttare a pieno le potenzialità di VMware Cloud ™ on AWS.
Amazon Elastic Container Service (Amazon ECS) è un servizio di gestione dei container altamente scalabile, che semplifica la gestione dei contenitori Docker attraverso un layer di orchestrazione per il controllo del deployment e del relativo lifecycle. In questa sessione presenteremo le principali caratteristiche del servizio, le architetture di riferimento per i differenti carichi di lavoro e i semplici passi necessari per poter velocemente migrare uno o più dei tuo container.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
3. Goals for today
• See real world use cases from 3 leading engineering
and scientific computing users
– Steve Philpott, CIO, HGST, A Western Digital Company
– Bill E. Williams, Director, The Aerospace Corporation
– Michael Steeves, Sr. Systems Engineer, Novartis
• Understand the motivations, strategies, lessons learned
in running HPC / Big Data workloads in the cloud
• See the varying scales and application types that run
well, including a 1.21 PetaFLOPS environment
4. Agenda
•
•
•
•
•
•
Introduction
Steve Philpott – Journey into Cloud
Bill Williams – Cloud Computing @ Aerospace
Michael Steeves – Accelerating Science
Spot, On-demand, & Other Production uses
Questions and answers
6. Cloud & Datacenter
Performance Enterprise
Founded in 2003 through the combination of the hard drive
businesses of IBM, the inventor of the hard drive, and
HGST, Ltd
PCIe
Enterprise SSD
(+3 acquisitions)
SAS
10K & 15K
HDDs
Acquired by Western Digital in 2012
More than 4,200 active worldwide patents
Headquartered in San Jose, California
Approximately 41,000 employees worldwide
Develops innovative, advanced hard disk drives, enterprise-class
solid state drives, external storage solutions and services
Ultrastar®
Capacity Enterprise
7200 RPM &
CoolSpin
HDDs
Ultrastar® &
MegaScale DC™
Delivers intelligent storage devices that tightly integrate hardware
and software to maximize solution performance
6
7. Zero to Cloud in 6+ Month
By 31 Oct 2013:
Cloud eMail – Microsoft Office365
April 2013
Cloud eMail archiving/eDiscovery
External SingleSignOn (off VPN)
Cloud File/Collaboration – BOX
Cloud CRM – Salesforce.com
Integrated to save files in BOX
Cloud–High Performance Computing
(HPC) on Amazon AWS
Cloud – Big Data Platform on Amazon AWS
7
8. Responding to the Changing Business Model
Where is our business model headed?
“New Age of Innovation” as a guide
N=1 Focus on Individual Customer Experience
R=G Resources are Global
Implications
–Increase in strategic partnering
–Need for high level of flexibility
–Leveraging external expertise
Use of the Cloud/SaaS aligns with
Virtual Business Model:
Variable cost model critically important
Lightweight, scalable services
Reduced up-front capital spend
Accelerated provisioning
Pay as you go
8
9. Paradigm Shift: Consumerization of IT
“I have better technology at home”
Consumer Web
A new paradigm in ease of use and reduced cost.
Consumer web has been driven by a series of
platforms – and these platforms are household brand
names today
When we use these platforms, it continually amazes
us – how easy, how consistent these platforms work
A new set of services: DRM to iTunes
Yet, our workplace applications are cumbersome, costly,
difficult to navigate and require extensive support
Workday, 2009
9
10. The Big Switch – The Box has Disappeared
The Transformation of Computing as we Know it.
Physical to Virtual/Digital move
– Do you really care which computer processed your last
Google search?
Efficiency
– Do not waste a CPU cycle or a byte of memory.
Building a 4-story building and only using the 1st floor
Utility: IT as a Service - Plug it in and get it
– Where the electricity industry has gone, Computing is following
– Computing shift is almost invisible to the end-user
DATA is the value to the Organization, not the “where”
1
11. Enabling the Virtual Organization
Reframing IT Away From Thinking of “The App”
Business Intelligence and Analytics
End-to-End Business Processes
Enterprise Data Management
New Computing Platforms
Strategic
Outsourcing
Software as a Service
(SaaS)
New IT Organizational Structures:
Support and Align to “New Business Model”
1
1
12. Creating an Innovation Playground:
Where to Start and How to Evolve
IT Supports Business Strategy
Executive Buy-In – CEO, CIO, InfoSec, etc
Reduce Cap-ex, Optimize DC usage
Build
Expertise
Implement
Outcome Defined
Knowledge
Play
Learn
Educate
• Team Involvement
• Conferences
• Vendor Briefing
• Expert Services
• Best Practices
Experiment
• Team Approach
• Hands-on approach
• Understand the value
proposition
• Understand constraints
Migrate
• Migrate dev/test
environments
• Migrate or
launch new apps
on the cloud
Embrace success
Showcase cost
savings
Build an enterprise
cloud strategy
Learn from each
experience
Expand accordingly
• Indentify app fit for cloud
computing
• Define new processes
• Collaborate with
other companies
12
Awareness
Understanding
Transition
Commitment
12
13. Multiple Opportunities to Leverage Amazon Web Services (AWS)
AWS: “ >5x the compute capacity than next
14 providers combined” – Gartner, Aug 2013
Access to massive compute and storage
Billed by the hour - only pay for what is used
HGST Japan Research Lab: Using AWS for higher
performance, lower cost, faster deployed solution vs. buying
huge on-site cluster
Develop AWS Competency
Many Opportunities: In-house and commercial HPCs are “cloud ready”
Provide Computing When Needed: Reduce capital investment & risk and increase flexibility
Faster Response to Business Needs: Rapid prototyping to pilot new IT capabilities with “PO
Process” ; setup users, allocate compute and storage in minutes, load apps and go
AWS provide a great option for disaster recovery for our “on-premise” clusters and storage
13
14. HGST’s Amazon HPC Platform
Case 3: Lube depletion in TAR (2D heat profile)
1.E+07
(300,000 atoms)
Atoms Dealing with
Basic Molecular Simulation
Large Scale Molecular Simulation for HDI
Top view
1.E+06
(Lube molecules spreading onto COC)
Case 3
5 ns
Case 1
1.E+05
1 ns
5 ns
Case 2
1.E+04
Relaxation time: 5 ns
Relaxation time: < 1 ns
1.E+03
0
100
200
300
400
Number of Core
500
600
Heat spot in TAR
36 nm
Molecular
Dynamics
Simulation
Read / Write
Magnetics
Electo –
Magnetic Fields
Mechanical
MAGLAND
Simulation
Application
CST Read / Write
Magnetics
Electo –
Magnetic Fields
Base HPC Platform
Scalable to thousands of
instances to support numerous
simultaneous simulations
Ansys
Commercial
LLG
Ansys
HFSS
Pre- and Post-Processing
Server Farms
New G2 Instances Add
Visualization Capabilities
14
15. Big Data’s “3 V’s”
Three “V’s” of Big Data
Best pragmatic
Volume
Velocity
•Data sources
•Data types
•Applications
Trends
Variety
•Data collected
•Analysis & metadata creation
•Data acquisition
•Analysis & action
Structured
Terabytes
Batch
Unstructured, Semi-Structured
& Structured
Petabytes & Exabytes
Real-Time & Streaming
Implications &
Opportunities
• Hardware and software optimization
• Architectural shifts: Scale-out systems, Distributed filesystems,
Tiered storage, Hadoop…
Key difference: data structure does not need
to be defined before loading
definition from
Snijders et al.
“Data sets so large
and complex that
they become
awkward to work
with using standard
tools and
techniques”
15
16. Data Sources
Big Data Platform
All raw parametric,
logistic, vintage, data
Parallelized
batch analytics
raw
extracts
Batch Analytics
Enriched
data
Slider
Wafer
Media
Substrate
Optimize/Reduce
Testing
End-to-End
Integrated
Data
.
.
.
SAP/DW’s
App-Specific
Views
Failure Screen
Tests
Proactive Drift
Identification
Field Data
Supplier
Ad hoc Analysis
Customer FA via
Field Data
HDD
HGA
Consumers
New High-Value
Parameters
SAS, Compellon or
other Predictive
Analytic Tools
Tableau, and
other tools
New Unified
EDW
16
17. Characteristics of a “Typical” Hadoop / Big Data Cluster
Hadoop handles large data volumes and reliability in the software tier
− Hadoop distributes data across cluster; uses replication to ensure data reliability
and fault tolerance.
Each machine in Hadoop cluster stores AND processes data; machines must do both well.
Processing sent directly to the machines storing the data.
Hadoop MapReduce
Compute Bound Operations
and Workloads
•
•
•
•
Clustering/Classification
Complex text mining
Natural-language processing
Feature extraction
Hadoop MapReduce
I/O Bound Operations
and Workloads
• Indexing
• Grouping
• Data importing and exporting
• Data movement and
transform
Big Data Solutions Must Support a Large Variety of
Compute and I/O Operations and Storage Needs …enter “the Cloud”
17
18. AWS Big Data Platform Storage Services
Block Storage for Elastic Computing
Optimized for Performance
SSD / 15K / 10K
Amazon
EBS
Highly Virtualized / SAN-Based
“Generic” Object Storage
Bulk of AWS Storage Today
Amazon
S3
Virtualized or Reserved Use
Server/Network-Based
Cold/Cool Storage
Amazon
Glacier
Lowest Cost Model for “least”
used data
3-5 hour Latency / Sequentialized
18
19. HGST’s Other Amazon Use Cases/Capabilities
Petabyte-Scale Data
Warehousing
“Between Glacier & S3”
Run Data Visualization
tools in AWS
Resource Tracking Tool
Includes Tableau
instance for reporting
and visualization
More and
more users
coming to IT
asking for how
to leverage
this new
compute
capability
19
20. We Are Just Starting with the Cloud
• Current Results From 6 month Effort
• Re-aligning Business Group Leadership
• Demands and Use To Grow And Accelerate
Cloud + HGST IT =
Strong Innovation and Business Partner
20
22. Introduction and Background
• IT Executive for the The Aerospace Corporation
•
•
(Aerospace)
Manage HPC compute and cloud resources for
the Aerospace corporate
Career path has taken me through end user
support, system administration, and enterprise
architecture
23. Agenda
•
•
•
•
•
•
•
•
Who is Aerospace?
High Performance Computing @ Aerospace
Services Provided
Cloud Motivation
Where are we today?
What makes this work?
Challenges
Lessons Learned
25. High Performance Computing @ Aerospace
• Allow engineers and scientists to focus on their
•
•
discipline and research
Reduce and eliminate complexity in using High
Performance Computing (HPC) resources
Supply and support centralized and networked
HPC resources
27. Cloud Motivation
•
•
•
•
•
•
Respond to an increasing and variable demand
Improve resource deployments and use
Enhance provisioning
Improve security posture
Improve disaster recovery posture
Greener
28. Where are we today?
•
•
•
•
Successfully established elastic clusters in AWS
GovCloud
– Workload runs include Monte Carlo and Array Simulations
Key features of the GovCloud clusters are
auto-scaling and on-demand computing
Compute instances are created as needed to meet job
computational requirements
Making strides towards mimicking internal clusters in
GovCloud
29. What makes this work?
• AWS GovCloud
– GovCloud is FedRAMP compliant
• Secure transport to and from Aerospace
– VPC provides an additional layer of security while data is in transit
• Cyclecomputing
– Cycle provides cluster auto-scaling
30. Lessons Learned
• Enhanced analytics and business intelligence
• Customer success stories
• Standard images
• Demonstrated operational “agility”
31. Lessons Learned
•
Domain space is dynamic
•
Expertise required
•
Layers of complexity
•
Ensuring data security (in hybrid deployment model)
32. Challenges
•
•
•
•
•
Establishing a cloud storage infrastructure
Determining appropriate bandwidth between
Aerospace and GovCloud
Library replication of internal systems
System integration with internal authentication
services
Insuring a seamless transition to hybrid services
33. What’s Next?
•
•
•
•
•
•
Expand offerings
Explore charge back
Explore “cloudifying” other HPC platforms
Track technology
Provide workload specific ad-hoc offerings
Provide surge capability for HPC resources
35. Novartis Institutes for BioMedical Research (NIBR)
Unique research strategy driven by patient needs
World-class research organization with about
6000 scientists globally
Intensifying focus on molecular pathways shared by
various diseases
Integration of clinical insights with mechanistic
understanding of disease
Research-to-Development transition redefined
through fast and rigorous “proof-of-concept” trials
Strategic alliances with academia and biotech
strengthen preclinical pipeline
36. Accelerating the Science
Requirements
Large Scale Computational Chemistry Simulation
Results in under a week
Ability to run multiple experiments “on-demand”
Challenges
Sustained access to 50000+ compute cores
Ability to monitor and re-launch jobs
No additional Capital Expenditure
Internal HPCC already running at capacity
Job Profile
Embarrassingly Parallel
CPU Bound
Low I/O, Memory and Network requirements
Virtual Screening
Target
Molecule
Compound
Molecule
binding
site
"Lock"
"Keys"
37. The Cloud: Flexible Science on Flexible Infrastructure
Engineering the right infrastructure for a workload:
Software runs the same job many times across instance types
Measures the throughput and determines the $ per job
Use the instances that provide the best scientific ROI
CC2 instance (Intel Xeon® ‘Sandy Bridge’) ran best for this
38. Super Computing in the Cloud
Metric
Compute Hours of Science
341,700 hours
Compute Days of Science
14,238 days
Compute Years of Science
39 years
AWS Instance Count-CC2
Count
10,600 instances
$44 Million infrastructure
10 million compounds screened
39 Drug Design years in 11 hours for a cost of …$4,232
3 compounds identified and synthesized for screening
39. Key Learnings/What’s Next?
Diversity of Life Sciences brings unique challenges
Spend the time analyzing and tuning
Flexibility, Scalability and Performance
Time to rethink and retool
Challenge the Science and the Scientist
Collaboration
Future plans
Chemical Universe : 166 Billion cpds (Extreme scale CPU)
Next Generation Sequencing in the Cloud (Extreme CPU, Mem, I/O)
“Disruptive” Technologies-Imaging (10x that of NGS!)
40. Using On-Demand and
Spot Instances together
When task durations are > than 1
hour or require multiple machines
(MPI) for long periods, then use ondemand
Shorter workloads work great for
Spot Instances
If you want a guaranteed end time,
use on-demand as well, so the
architecture looks like…
41. User
Scale from 150 - 150,000+ cores
CycleCloud Deploys Secured, Auto-scaled HPC Clusters
HPC Cluster
Load-based Spot bidding
On-Demand Execute Nodes
(Guaranteed finish)
Check job load
Calculate ideal HPC cluster
Legacy
Internal
HPC
Shared FS
Spot Instance Execute Nodes
(auto-started & auto-stopped
calculation is faster/cheaper)
Properly price the bids
Manage Spot Instance loss
FS /
S3
HPC Orchestration to
Handle Spot Instance Bid & Loss
42. Other Production use cases
•
•
•
•
•
•
•
Sequencing, Genomics, Life Sciences
MPI workloads for FEA, CFD, energy, utilities
MATLAB and R applications for stats/modeling
Win HPC Server cluster for finance
Heat transfer and other FEA
Insurance risk management
Rendering/VFX
43. Designing Solar Materials
The Challenge is efficiency
Need to efficiently turn photons from the sun to Electricity
The number of possible materials is limitless:
• Need to separate the right compounds from the useless ones
• If the 20th century was the century of silicon, the 21st will be
all organic
How do we find the right material out of 205,000
without spending the entire 21st century looking for it?
EMBARGOED until Nov. 12, 2013 8 a.m. EST
51. Question and Answer
How does utility HPC apply to your organization?
Follow us: @cyclecomputing, @jasonastowe
Come to Cycle’s booth: #1112
We’re hiring jointheteam@cyclecomputing.com
52. Please give us your feedback on this
presentation
BDT212
As a thank you, we will select prize
winners daily for completed surveys!