Lustre Releases Update from LAD'14
•
0 likes•1,306 views
In this video from the LAD'14 Lustre Administrators and Developers Conference, Peter Jones from Intel presents: Lustre Releases. Learn more: http://www.eofs.eu/?id=lad14 Watch the video presentation: http://wp.me/p3RLHQ-d1q
Report
Share
Report
Share
Download to read offline
Recommended
Dell Lustre Storage Architecture Presentation - MBUG 2016
Overview of the architecture, and benefits of Dell HPC Storage with Intel EE Lustre in High Performance Computing and Big Science workloads.
Presented by Andrew Underwood at the Melbourne Big Data User Group - January 2016.
Lustre is a trademark of Seagate Technology.
Performance Comparison of Intel Enterprise Edition Lustre and HDFS for MapRed...
In this deck from the LAD'14 Conference in Reims, Rekha Singhal from Tata Consultancy Services presents: Performance Comparison of Intel Enterprise Edition Lustre and HDFS for MapReduce Application.
Learn more: http://insidehpc.com/lad14-video-gallery/
Watch the video presentation: http://inside-bigdata.com/2014/09/29/performance-comparison-intel-enterprise-edition-lustre-hdfs-mapreduce/
The Importance of Fast, Scalable Storage for Today’s HPC
Today, data drives discovery. And discoveries create are key to creating sustained advantages. The better your critical workflows are able to create and access data – the better you’ll be able to discover new, innovative solutions to important problems, or to create entirely new products. More than ever before, data intensive applications need the sustained performance and virtually unlimited scalability that only parallel storage software delivers.
Designed for maximum performance and scale, storage solutions powered by Lustre software deliver the performance at scale to meet today’s storage requirements. As the most widely used parallel storage system for HPC, Lustre-powered storage is the ideal storage foundation.
But scalable performance storage by itself only solves half the problem. Today’s users expect storage solutions that deliver sustained performance, scale upward to near limitless capacities, and are simple to install and manage. Intel(r) Enterprise Edition for Lustre* software combines the straight line speed and scale of Lustre with the bottom line need for lowered management complexity and cost.
As the recognized leaders in the development and support of the Lustre file system, Intel has the expertise to make storage solutions for data intensive applications faster, smarter and easier.
Blazing Fast Lustre Storage
This document discusses the NetApp E5500 storage solution for Lustre file systems. It provides three key points:
1) The NetApp E5500 is designed to meet the demands of large Lustre file systems including supporting over 100TB of storage, 100,000 clients, and independent scaling of clients, storage, and bandwidth.
2) Lustre is an open source parallel file system used on over 60% of the world's largest supercomputers that separates data from metadata to deliver scale and performance.
3) Test results show the E5500 can deliver over 7,200 sustained MBps of throughput from compute nodes to a 250TB Lustre file system, demonstrating its
Accelerating analytics workloads with Alluxio data orchestration and Intel® O...
The document discusses test results from running the TPC-DS benchmark on Spark SQL with different Alluxio caching layers: DRAM, Intel Optane persistent memory (PMem) using Storage over App Direct mode, and Intel Optane SSDs. PMem and Optane SSD caching completed more queries than DRAM-only caching and had significantly faster query times. While PMem and Optane SSD caching performed similarly in CPU usage and hardware metrics, PMem caching had some advantages like higher caching capacity and lower query times for some workloads. More analysis is needed to understand Alluxio-specific metrics and performance in cloud environments.
Backup Options for IBM PureData for Analytics powered by Netezza
This document discusses backup options for IBM PureData System for Analytics powered by Netezza. It describes using either a filesystem approach backing up metadata and databases to external storage, or using external backup software. When using external backup software, the document recommends IBM Tivoli Storage Manager and describes backup architectures using a TSM proxy node or LAN-free configuration. It also provides best practices like running multiple backup streams in parallel.
Database server comparison: Dell PowerEdge R630 vs. Lenovo ThinkServer RD550
We tested the OLTP performance of a 1U Dell PowerEdge R630, powered by Intel Xeon processors E5-2660 v3, running Microsoft Hyper-V and virtual machines running SQL Server 2014, and compared it to that of the Lenovo ThinkServer RD550 running the same software. For each server, we selected the maximum SATA SSD count that was configurable for each model. The Dell PowerEdge R630 outperformed the Lenovo ThinkServer RD550 by 14.9 percent and offered more than one and a half times the storage space for data in our configuration.
By selecting a server that handles more orders per minute and offers significantly more storage capacity potential than the competition, you get a not only faster, efficient experience for your database users, but also have the scaling potential for your storage needs ahead of your business growing.
INTEL® XEON® SCALABLE PROCESSORS
The innovative Intel® Xeon® Scalable processors are architected to provide the
foundation for mission-critical workloads. The new Intel® Xeon® Platinum and Gold
processors are optimized to deliver exceptionally fast performance and high reliability
for robust business continuity.
Recommended
Dell Lustre Storage Architecture Presentation - MBUG 2016
Overview of the architecture, and benefits of Dell HPC Storage with Intel EE Lustre in High Performance Computing and Big Science workloads.
Presented by Andrew Underwood at the Melbourne Big Data User Group - January 2016.
Lustre is a trademark of Seagate Technology.
Performance Comparison of Intel Enterprise Edition Lustre and HDFS for MapRed...
In this deck from the LAD'14 Conference in Reims, Rekha Singhal from Tata Consultancy Services presents: Performance Comparison of Intel Enterprise Edition Lustre and HDFS for MapReduce Application.
Learn more: http://insidehpc.com/lad14-video-gallery/
Watch the video presentation: http://inside-bigdata.com/2014/09/29/performance-comparison-intel-enterprise-edition-lustre-hdfs-mapreduce/
The Importance of Fast, Scalable Storage for Today’s HPC
Today, data drives discovery. And discoveries create are key to creating sustained advantages. The better your critical workflows are able to create and access data – the better you’ll be able to discover new, innovative solutions to important problems, or to create entirely new products. More than ever before, data intensive applications need the sustained performance and virtually unlimited scalability that only parallel storage software delivers.
Designed for maximum performance and scale, storage solutions powered by Lustre software deliver the performance at scale to meet today’s storage requirements. As the most widely used parallel storage system for HPC, Lustre-powered storage is the ideal storage foundation.
But scalable performance storage by itself only solves half the problem. Today’s users expect storage solutions that deliver sustained performance, scale upward to near limitless capacities, and are simple to install and manage. Intel(r) Enterprise Edition for Lustre* software combines the straight line speed and scale of Lustre with the bottom line need for lowered management complexity and cost.
As the recognized leaders in the development and support of the Lustre file system, Intel has the expertise to make storage solutions for data intensive applications faster, smarter and easier.
Blazing Fast Lustre Storage
This document discusses the NetApp E5500 storage solution for Lustre file systems. It provides three key points:
1) The NetApp E5500 is designed to meet the demands of large Lustre file systems including supporting over 100TB of storage, 100,000 clients, and independent scaling of clients, storage, and bandwidth.
2) Lustre is an open source parallel file system used on over 60% of the world's largest supercomputers that separates data from metadata to deliver scale and performance.
3) Test results show the E5500 can deliver over 7,200 sustained MBps of throughput from compute nodes to a 250TB Lustre file system, demonstrating its
Accelerating analytics workloads with Alluxio data orchestration and Intel® O...
The document discusses test results from running the TPC-DS benchmark on Spark SQL with different Alluxio caching layers: DRAM, Intel Optane persistent memory (PMem) using Storage over App Direct mode, and Intel Optane SSDs. PMem and Optane SSD caching completed more queries than DRAM-only caching and had significantly faster query times. While PMem and Optane SSD caching performed similarly in CPU usage and hardware metrics, PMem caching had some advantages like higher caching capacity and lower query times for some workloads. More analysis is needed to understand Alluxio-specific metrics and performance in cloud environments.
Backup Options for IBM PureData for Analytics powered by Netezza
This document discusses backup options for IBM PureData System for Analytics powered by Netezza. It describes using either a filesystem approach backing up metadata and databases to external storage, or using external backup software. When using external backup software, the document recommends IBM Tivoli Storage Manager and describes backup architectures using a TSM proxy node or LAN-free configuration. It also provides best practices like running multiple backup streams in parallel.
Database server comparison: Dell PowerEdge R630 vs. Lenovo ThinkServer RD550
We tested the OLTP performance of a 1U Dell PowerEdge R630, powered by Intel Xeon processors E5-2660 v3, running Microsoft Hyper-V and virtual machines running SQL Server 2014, and compared it to that of the Lenovo ThinkServer RD550 running the same software. For each server, we selected the maximum SATA SSD count that was configurable for each model. The Dell PowerEdge R630 outperformed the Lenovo ThinkServer RD550 by 14.9 percent and offered more than one and a half times the storage space for data in our configuration.
By selecting a server that handles more orders per minute and offers significantly more storage capacity potential than the competition, you get a not only faster, efficient experience for your database users, but also have the scaling potential for your storage needs ahead of your business growing.
INTEL® XEON® SCALABLE PROCESSORS
The innovative Intel® Xeon® Scalable processors are architected to provide the
foundation for mission-critical workloads. The new Intel® Xeon® Platinum and Gold
processors are optimized to deliver exceptionally fast performance and high reliability
for robust business continuity.
Technical Report NetApp Clustered Data ONTAP 8.2: An Introduction
This technical report is an introduction to the architecture and key customer benefits of NetApp clustered Data ONTAP 8.2.
Make sense of important data faster with AWS EC2 M6i instances
Featuring 3rd Gen Intel Xeon Scalable processors, M6i instances sped up data analysis over M5 instances with older processors
HDFS Analysis for Small Files
This document summarizes a presentation about analyzing small files in HDFS clusters. It outlines the problems small files can cause, such as inefficient data access and slower jobs. It then describes the architecture of the small files analysis solution, which processes the HDFS fsimage to attribute and aggregate file information. This information is stored and used to power dashboards showing metrics like small file counts and distributions over time. Future work includes improving performance and developing a customizable compaction utility.
Get insight from document-based distributed MongoDB databases sooner and have...
Get insight from document-based distributed MongoDB databases sooner and have...Principled Technologies
With additional drive bays and 2nd Generation Intel Xeon Scalable processors, Dell EMC PowerEdge R640 servers handled more Yahoo Cloud Serving Benchmark (YCSB) operations per second than previous-generation servers and handled them more efficientlyIntel and Red Hat: Enhancing OpenStack for Enterprise Deployment
The document discusses Intel and Red Hat's joint efforts to enhance OpenStack for enterprise deployment. Some key points:
- They are addressing foundational capabilities for OpenStack like security, high performance, and efficiency to help drive enterprise adoption.
- Intel is contributing to many OpenStack projects and working with Red Hat and the ecosystem.
- Red Hat OpenStack Platform incorporates Intel technologies for capabilities like trusted compute pools, intelligent scheduling, and software-defined networking.
- Upcoming releases aim to further optimize areas like storage utilization, power management, and network performance for networking functions.
Flexible and Fast Storage for Deep Learning with Alluxio
This document discusses how Alluxio provides fast and flexible storage for deep learning workloads. It summarizes Alluxio's capabilities to accelerate data processing and machine learning workflows by enabling data to be stored, cached, and processed directly in memory across distributed environments. Alluxio uses a unified namespace and intelligent caching to provide high-speed data access to remote data sources and overcome storage bottlenecks.
Big Data Platform Industrialization
The document outlines Renault's big data initiatives from 2014-2016, including:
1. Starting with a big data sandbox in 2014 using an old HPC infrastructure for data exploration.
2. Implementing a DataLab in 2015 with a new HP infrastructure and establishing a first level of industrialization while improving data protection.
3. Creating a big data platform in 2016 to industrialize hosting both proofs of concept and production projects while ensuring data protection.
Advancing GPU Analytics with RAPIDS Accelerator for Spark and Alluxio
This document discusses accelerating Apache Spark workloads using RAPIDS Accelerator for Spark and Alluxio. It provides an introduction to RAPIDS Accelerator for Spark, shows significant performance gains over CPU-only Spark, and discusses combining GPU acceleration with Alluxio for optimized performance and cost on cloud datasets. Configuration options for RAPIDS and Alluxio are also covered.
Best Practices for Using Alluxio with Apache Spark with Gene Pang
Alluxio, formerly Tachyon, is a memory speed virtual distributed storage system and leverages memory for storing data and accelerating access to data in different storage systems. Many organizations and deployments use Alluxio with Apache Spark, and some of them scale out to over PB’s of data. Alluxio can enable Spark to be even more effective, in both on-premise deployments and public cloud deployments. Alluxio bridges Spark applications with various storage systems and further accelerates data intensive applications. In this talk, we briefly introduce Alluxio, and present different ways how Alluxio can help Spark jobs. We discuss best practices of using Alluxio with Spark, including RDDs and DataFrames, as well as on-premise deployments and public cloud deployments.
Comparing Dell Compellent network-attached storage to an industry-leading NAS...
Comparing Dell Compellent network-attached storage to an industry-leading NAS...Principled Technologies
A flexible NAS solution addresses many organizational challenges from server backup to hosting production applications and databases. Advanced NAS solutions such as the Intel Xeon processor-based Dell Compellent FS8600 NAS provide flexibility and scalability, allowing various use options as well as drive options throughout its lifecycle. This scale and flexibility enables an organization to alleviate performance bottlenecks anywhere in the organization simply by reallocating or adding more disk resources.
We found that the Intel Xeon processor-based Dell Compellent FS8600 NAS solution backed up a small-file corpus up to 15.9 percent faster and a large-file corpus up to 17.1 percent faster than a similarly configured, industry-leading NAS solution. This means that selecting the Dell Compellent FS8600 NAS has the potential to help optimize an organization’s infrastructure.
White Paper: Using Perforce 'Attributes' for Managing Game Asset Metadata
Perforce attributes are used to organize and manipulate game assets. Attributes store metadata like categorization and dependency information. A local SQLite database caches attribute information for faster searching. Integrating assets between branches is challenging since attributes cannot be merged like text files.
Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics
Alluxio Austin Meetup
Aug 15, 2019
Speaker: Bin Fan
Apache Spark and Alluxio are cousin open source projects that originated from UC Berkeley’s AMPLab. Running Spark with Alluxio is a popular stack particularly for hybrid environments. In this session, I will briefly introduce Apache Spark and Alluxio, share the top ten tips for performance tuning for real-world workloads, and demo Alluxio with Spark.
HDFS Erasure Coding in Action
This document discusses HDFS Erasure Coding and its usage at Yahoo Japan. It begins with an overview of erasure coding, how it is implemented in HDFS, and compares it to replication. Test results show the write performance is lower for erasure coding while read performance is similar. Yahoo Japan uses erasure coding for cold weblog data, reducing storage costs by 65% compared to replication. Future plans include supporting additional codecs and features to provide more usability.
Scalability: Lenovo ThinkServer RD540 system and Lenovo ThinkServer SA120 sto...
Scalability: Lenovo ThinkServer RD540 system and Lenovo ThinkServer SA120 sto...Principled Technologies
Enterprises and SMBs need servers that can provide reliable performance with the ability to scale out to match growth. The Lenovo ThinkServer RD540 and the ThinkServer SA120 DAS array can run transactional applications such as Microsoft Exchange Server while providing scalable storage to support these critical workloads. We found that in the HDD configuration, the ThinkServer RD540 and ThinkServer SA120 DAS device provided support for 3,800 Exchange users. When we added just two Intel 400GB SSDs as a CacheCade volume, the ThinkServer RD540 and ThinkServer SA120 not only supported 5,300 users—a 39.5 percent increase—but did so while improving response time 33.9 percent.
How to Protect Big Data in a Containerized Environment
Every enterprise spends significant resources to protect its data. This is especially true in the case of big data, since some of this data may include sensitive or confidential customer and financial information. Common methods for protecting data include permissions and access controls as well as the encryption of data at rest and in flight.
The Hadoop community has recently rolled out Transparent Data Encryption (TDE) support in HDFS. Transparent Data Encryption refers to the process whereby data is transparently encrypted by the big data application writing the data; it is not decrypted again until it is accessed by another application. The data is encrypted during its entire lifespan—in transit and at rest—except when it is being specifically accessed by a processing application.
TDE is an excellent approach for protecting data stored in data lakes built on the latest versions of HDFS. However, it does have its challenges and limitations. Systems that want to use TDE require tight integration with enterprise-wide Kerberos Key Distribution Center (KDC) services and Key Management Systems (KMS). This integration isn’t easy to set up or maintain. These issues can be even more challenging in a virtualized or containerized environment where one Kerberos realm may be used to secure the big data compute cluster and a different Kerberos realm may be used to secure the HDFS filesystem accessed by this cluster.
BlueData has developed significant expertise in configuring, managing, and optimizing access to TDE-protected HDFS. This session at the Strata Data Conference in March 2018 (by Thomas Phelan, co-founder and chief architect at BlueData) offers a detailed overview of how transparent data encryption works with HDFS, with a particular focus on containerized environments.
You’ll learn how HDFS TDE is configured and maintained in an environment where many big data frameworks run simultaneously (e.g., in a hybrid cloud architecture using Docker containers). Moreover, you’ll learn how KDC credentials can be managed in a Kerberos cross-realm environment to provide data scientists and analysts with the greatest flexibility in accessing data while maintaining complete enterprise-grade data security.
https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63763
Software-Defined Storage (SDS)
An SDS (software-defined storage) refers to a software controller that is used for managing and virtualizing a physical storage for the purpose of controlling the way in which data is stored.
Consolidating database servers with Lenovo ThinkServer RD630
Populating your data center with new, more powerful and energy efficient servers can deliver numerous benefits to your organization. By consolidating multiple older servers onto a new platform, you can save in the areas of data center space and port costs, management costs, and power and cooling costs.
In our tests, we found that the Lenovo ThinkServer RD630 could consolidate the workloads of three HP ProLiant DL385 G5 servers, while increasing overall performance by 82.6 percent and reducing power consumption by 58.8 percent, making the ThinkServer RD630 an excellent choice to reduce the costs associated with running your data center.
EMC Isilon Best Practices for Hadoop Data Storage
This white paper describes the best practices for setting up and managing the HDFS service on an Isilon cluster to optimize data storage for Hadoop analytics.
Dell Poweredge FX Infographic
Bridge traditional & new IT Workloads on one platform...
A wave of new, modern workloads is forcing companies to find a bridge between current infrastructure and future investment without sacrificing cost, capability or confidence. By converging hardware, management and workloads, the PowerEdge FX architecture allows customers to precisely tailor, quickly deploy and easily manage a diverse set of applications using a common set of modular IT building blocks.
The PowerEdge FX architecture takes the density and modularity of blades, adds the cost efficiency and manageability of racks, and combines that with modular storage, a flexible fabric and future-ready server nodes. All engineered to provide IT flexibility to shift and scale as requirements change in the future.
For more information:
http://www.dell.com/FX
Deploying Big-Data-as-a-Service (BDaaS) in the Enterprise
This document summarizes a presentation about deploying Big Data as a Service (BDaaS) in the enterprise. It discusses how BDaaS can address conflicting needs of data scientists wanting flexibility and IT wanting control. It defines different types of BDaaS and requirements for enterprise deployment such as multi-tenancy, security, and application support. The presentation covers design decisions for BDaaS including running Hadoop/Spark unmodified using containers for isolation. It provides details on the implementation including network architecture, storage, and image management. It also discusses performance testing results and demos the BDaaS platform.
Cycle Computing Record-breaking Petascale HPC Run
In this slidecast, Jason Stowe from Cycle Computing describes the company's recent record-breaking Petascale CycleCloud HPC production run.
"For this big workload, a 156,314-core CycleCloud behemoth spanning 8 AWS regions, totaling 1.21 petaFLOPS (RPeak, not RMax) of aggregate compute power, to simulate 205,000 materials, crunched 264 compute years in only 18 hours. Thanks to Cycle's software and Amazon's Spot Instances, a supercomputing environment worth $68M if you had bought it, ran 2.3 Million hours of material science, approximately 264 compute-years, of simulation in only 18 hours, cost only $33,000, or $0.16 per molecule."
Learn more: http://blog.cyclecomputing.com/2013/11/back-to-the-future-121-petaflopsrpeak-156000-core-cyclecloud-hpc-runs-264-years-of-materials-science.html
Watch the video presentation: http://wp.me/p3RLHQ-aO9
DDN: Massively-Scalable Platforms and Solutions Engineered for the Big Data a...
In this talk from the DDN User Group at ISC’13, James Coomer from DataDirect Networks presents: Massively-Scalable Platforms and Solutions Engineered for the Big Data and Cloud Era.
Watch the presentation here: http://insidehpc.com/2013/06/26/video-james-coomer-keynotes-ddn-user-group-at-isc13/
More Related Content
What's hot
Technical Report NetApp Clustered Data ONTAP 8.2: An Introduction
This technical report is an introduction to the architecture and key customer benefits of NetApp clustered Data ONTAP 8.2.
Make sense of important data faster with AWS EC2 M6i instances
Featuring 3rd Gen Intel Xeon Scalable processors, M6i instances sped up data analysis over M5 instances with older processors
HDFS Analysis for Small Files
This document summarizes a presentation about analyzing small files in HDFS clusters. It outlines the problems small files can cause, such as inefficient data access and slower jobs. It then describes the architecture of the small files analysis solution, which processes the HDFS fsimage to attribute and aggregate file information. This information is stored and used to power dashboards showing metrics like small file counts and distributions over time. Future work includes improving performance and developing a customizable compaction utility.
Get insight from document-based distributed MongoDB databases sooner and have...
Get insight from document-based distributed MongoDB databases sooner and have...Principled Technologies
With additional drive bays and 2nd Generation Intel Xeon Scalable processors, Dell EMC PowerEdge R640 servers handled more Yahoo Cloud Serving Benchmark (YCSB) operations per second than previous-generation servers and handled them more efficientlyIntel and Red Hat: Enhancing OpenStack for Enterprise Deployment
The document discusses Intel and Red Hat's joint efforts to enhance OpenStack for enterprise deployment. Some key points:
- They are addressing foundational capabilities for OpenStack like security, high performance, and efficiency to help drive enterprise adoption.
- Intel is contributing to many OpenStack projects and working with Red Hat and the ecosystem.
- Red Hat OpenStack Platform incorporates Intel technologies for capabilities like trusted compute pools, intelligent scheduling, and software-defined networking.
- Upcoming releases aim to further optimize areas like storage utilization, power management, and network performance for networking functions.
Flexible and Fast Storage for Deep Learning with Alluxio
This document discusses how Alluxio provides fast and flexible storage for deep learning workloads. It summarizes Alluxio's capabilities to accelerate data processing and machine learning workflows by enabling data to be stored, cached, and processed directly in memory across distributed environments. Alluxio uses a unified namespace and intelligent caching to provide high-speed data access to remote data sources and overcome storage bottlenecks.
Big Data Platform Industrialization
The document outlines Renault's big data initiatives from 2014-2016, including:
1. Starting with a big data sandbox in 2014 using an old HPC infrastructure for data exploration.
2. Implementing a DataLab in 2015 with a new HP infrastructure and establishing a first level of industrialization while improving data protection.
3. Creating a big data platform in 2016 to industrialize hosting both proofs of concept and production projects while ensuring data protection.
Advancing GPU Analytics with RAPIDS Accelerator for Spark and Alluxio
This document discusses accelerating Apache Spark workloads using RAPIDS Accelerator for Spark and Alluxio. It provides an introduction to RAPIDS Accelerator for Spark, shows significant performance gains over CPU-only Spark, and discusses combining GPU acceleration with Alluxio for optimized performance and cost on cloud datasets. Configuration options for RAPIDS and Alluxio are also covered.
Best Practices for Using Alluxio with Apache Spark with Gene Pang
Alluxio, formerly Tachyon, is a memory speed virtual distributed storage system and leverages memory for storing data and accelerating access to data in different storage systems. Many organizations and deployments use Alluxio with Apache Spark, and some of them scale out to over PB’s of data. Alluxio can enable Spark to be even more effective, in both on-premise deployments and public cloud deployments. Alluxio bridges Spark applications with various storage systems and further accelerates data intensive applications. In this talk, we briefly introduce Alluxio, and present different ways how Alluxio can help Spark jobs. We discuss best practices of using Alluxio with Spark, including RDDs and DataFrames, as well as on-premise deployments and public cloud deployments.
Comparing Dell Compellent network-attached storage to an industry-leading NAS...
Comparing Dell Compellent network-attached storage to an industry-leading NAS...Principled Technologies
A flexible NAS solution addresses many organizational challenges from server backup to hosting production applications and databases. Advanced NAS solutions such as the Intel Xeon processor-based Dell Compellent FS8600 NAS provide flexibility and scalability, allowing various use options as well as drive options throughout its lifecycle. This scale and flexibility enables an organization to alleviate performance bottlenecks anywhere in the organization simply by reallocating or adding more disk resources.
We found that the Intel Xeon processor-based Dell Compellent FS8600 NAS solution backed up a small-file corpus up to 15.9 percent faster and a large-file corpus up to 17.1 percent faster than a similarly configured, industry-leading NAS solution. This means that selecting the Dell Compellent FS8600 NAS has the potential to help optimize an organization’s infrastructure.
White Paper: Using Perforce 'Attributes' for Managing Game Asset Metadata
Perforce attributes are used to organize and manipulate game assets. Attributes store metadata like categorization and dependency information. A local SQLite database caches attribute information for faster searching. Integrating assets between branches is challenging since attributes cannot be merged like text files.
Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics
Alluxio Austin Meetup
Aug 15, 2019
Speaker: Bin Fan
Apache Spark and Alluxio are cousin open source projects that originated from UC Berkeley’s AMPLab. Running Spark with Alluxio is a popular stack particularly for hybrid environments. In this session, I will briefly introduce Apache Spark and Alluxio, share the top ten tips for performance tuning for real-world workloads, and demo Alluxio with Spark.
HDFS Erasure Coding in Action
This document discusses HDFS Erasure Coding and its usage at Yahoo Japan. It begins with an overview of erasure coding, how it is implemented in HDFS, and compares it to replication. Test results show the write performance is lower for erasure coding while read performance is similar. Yahoo Japan uses erasure coding for cold weblog data, reducing storage costs by 65% compared to replication. Future plans include supporting additional codecs and features to provide more usability.
Scalability: Lenovo ThinkServer RD540 system and Lenovo ThinkServer SA120 sto...
Scalability: Lenovo ThinkServer RD540 system and Lenovo ThinkServer SA120 sto...Principled Technologies
Enterprises and SMBs need servers that can provide reliable performance with the ability to scale out to match growth. The Lenovo ThinkServer RD540 and the ThinkServer SA120 DAS array can run transactional applications such as Microsoft Exchange Server while providing scalable storage to support these critical workloads. We found that in the HDD configuration, the ThinkServer RD540 and ThinkServer SA120 DAS device provided support for 3,800 Exchange users. When we added just two Intel 400GB SSDs as a CacheCade volume, the ThinkServer RD540 and ThinkServer SA120 not only supported 5,300 users—a 39.5 percent increase—but did so while improving response time 33.9 percent.
How to Protect Big Data in a Containerized Environment
Every enterprise spends significant resources to protect its data. This is especially true in the case of big data, since some of this data may include sensitive or confidential customer and financial information. Common methods for protecting data include permissions and access controls as well as the encryption of data at rest and in flight.
The Hadoop community has recently rolled out Transparent Data Encryption (TDE) support in HDFS. Transparent Data Encryption refers to the process whereby data is transparently encrypted by the big data application writing the data; it is not decrypted again until it is accessed by another application. The data is encrypted during its entire lifespan—in transit and at rest—except when it is being specifically accessed by a processing application.
TDE is an excellent approach for protecting data stored in data lakes built on the latest versions of HDFS. However, it does have its challenges and limitations. Systems that want to use TDE require tight integration with enterprise-wide Kerberos Key Distribution Center (KDC) services and Key Management Systems (KMS). This integration isn’t easy to set up or maintain. These issues can be even more challenging in a virtualized or containerized environment where one Kerberos realm may be used to secure the big data compute cluster and a different Kerberos realm may be used to secure the HDFS filesystem accessed by this cluster.
BlueData has developed significant expertise in configuring, managing, and optimizing access to TDE-protected HDFS. This session at the Strata Data Conference in March 2018 (by Thomas Phelan, co-founder and chief architect at BlueData) offers a detailed overview of how transparent data encryption works with HDFS, with a particular focus on containerized environments.
You’ll learn how HDFS TDE is configured and maintained in an environment where many big data frameworks run simultaneously (e.g., in a hybrid cloud architecture using Docker containers). Moreover, you’ll learn how KDC credentials can be managed in a Kerberos cross-realm environment to provide data scientists and analysts with the greatest flexibility in accessing data while maintaining complete enterprise-grade data security.
https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63763
Software-Defined Storage (SDS)
An SDS (software-defined storage) refers to a software controller that is used for managing and virtualizing a physical storage for the purpose of controlling the way in which data is stored.
Consolidating database servers with Lenovo ThinkServer RD630
Populating your data center with new, more powerful and energy efficient servers can deliver numerous benefits to your organization. By consolidating multiple older servers onto a new platform, you can save in the areas of data center space and port costs, management costs, and power and cooling costs.
In our tests, we found that the Lenovo ThinkServer RD630 could consolidate the workloads of three HP ProLiant DL385 G5 servers, while increasing overall performance by 82.6 percent and reducing power consumption by 58.8 percent, making the ThinkServer RD630 an excellent choice to reduce the costs associated with running your data center.
EMC Isilon Best Practices for Hadoop Data Storage
This white paper describes the best practices for setting up and managing the HDFS service on an Isilon cluster to optimize data storage for Hadoop analytics.
Dell Poweredge FX Infographic
Bridge traditional & new IT Workloads on one platform...
A wave of new, modern workloads is forcing companies to find a bridge between current infrastructure and future investment without sacrificing cost, capability or confidence. By converging hardware, management and workloads, the PowerEdge FX architecture allows customers to precisely tailor, quickly deploy and easily manage a diverse set of applications using a common set of modular IT building blocks.
The PowerEdge FX architecture takes the density and modularity of blades, adds the cost efficiency and manageability of racks, and combines that with modular storage, a flexible fabric and future-ready server nodes. All engineered to provide IT flexibility to shift and scale as requirements change in the future.
For more information:
http://www.dell.com/FX
Deploying Big-Data-as-a-Service (BDaaS) in the Enterprise
This document summarizes a presentation about deploying Big Data as a Service (BDaaS) in the enterprise. It discusses how BDaaS can address conflicting needs of data scientists wanting flexibility and IT wanting control. It defines different types of BDaaS and requirements for enterprise deployment such as multi-tenancy, security, and application support. The presentation covers design decisions for BDaaS including running Hadoop/Spark unmodified using containers for isolation. It provides details on the implementation including network architecture, storage, and image management. It also discusses performance testing results and demos the BDaaS platform.
What's hot (20)
Technical Report NetApp Clustered Data ONTAP 8.2: An Introduction
Technical Report NetApp Clustered Data ONTAP 8.2: An Introduction
Make sense of important data faster with AWS EC2 M6i instances
Make sense of important data faster with AWS EC2 M6i instances
Get insight from document-based distributed MongoDB databases sooner and have...
Get insight from document-based distributed MongoDB databases sooner and have...
Intel and Red Hat: Enhancing OpenStack for Enterprise Deployment
Intel and Red Hat: Enhancing OpenStack for Enterprise Deployment
Flexible and Fast Storage for Deep Learning with Alluxio
Flexible and Fast Storage for Deep Learning with Alluxio
Advancing GPU Analytics with RAPIDS Accelerator for Spark and Alluxio
Advancing GPU Analytics with RAPIDS Accelerator for Spark and Alluxio
Best Practices for Using Alluxio with Apache Spark with Gene Pang
Best Practices for Using Alluxio with Apache Spark with Gene Pang
Comparing Dell Compellent network-attached storage to an industry-leading NAS...
Comparing Dell Compellent network-attached storage to an industry-leading NAS...
White Paper: Using Perforce 'Attributes' for Managing Game Asset Metadata
White Paper: Using Perforce 'Attributes' for Managing Game Asset Metadata
Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics
Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics
Scalability: Lenovo ThinkServer RD540 system and Lenovo ThinkServer SA120 sto...
Scalability: Lenovo ThinkServer RD540 system and Lenovo ThinkServer SA120 sto...
How to Protect Big Data in a Containerized Environment
How to Protect Big Data in a Containerized Environment
Consolidating database servers with Lenovo ThinkServer RD630
Consolidating database servers with Lenovo ThinkServer RD630
Deploying Big-Data-as-a-Service (BDaaS) in the Enterprise
Deploying Big-Data-as-a-Service (BDaaS) in the Enterprise
Viewers also liked
Cycle Computing Record-breaking Petascale HPC Run
In this slidecast, Jason Stowe from Cycle Computing describes the company's recent record-breaking Petascale CycleCloud HPC production run.
"For this big workload, a 156,314-core CycleCloud behemoth spanning 8 AWS regions, totaling 1.21 petaFLOPS (RPeak, not RMax) of aggregate compute power, to simulate 205,000 materials, crunched 264 compute years in only 18 hours. Thanks to Cycle's software and Amazon's Spot Instances, a supercomputing environment worth $68M if you had bought it, ran 2.3 Million hours of material science, approximately 264 compute-years, of simulation in only 18 hours, cost only $33,000, or $0.16 per molecule."
Learn more: http://blog.cyclecomputing.com/2013/11/back-to-the-future-121-petaflopsrpeak-156000-core-cyclecloud-hpc-runs-264-years-of-materials-science.html
Watch the video presentation: http://wp.me/p3RLHQ-aO9
DDN: Massively-Scalable Platforms and Solutions Engineered for the Big Data a...
In this talk from the DDN User Group at ISC’13, James Coomer from DataDirect Networks presents: Massively-Scalable Platforms and Solutions Engineered for the Big Data and Cloud Era.
Watch the presentation here: http://insidehpc.com/2013/06/26/video-james-coomer-keynotes-ddn-user-group-at-isc13/
AWS Customer Presentation - Cycle Computing - AWS Summit 2012 - NYC
The document discusses how CycleCloud, a utility supercomputing platform, was able to run a massive virtual screen of 21 million compounds on a 50,000-core cluster on AWS in just 3 hours for $4828.85, providing a cost-effective alternative to building and maintaining an internal high performance computing cluster that would have cost over $20 million and taken years to complete the same task. The ability to access massive parallel computing resources on demand allows researchers to tackle problems that were previously impossible due to limitations of internal computing infrastructure.
Cycle Cloud 70,000 Core AWS Cluster for HGST
In this slidecast, Jason Stowe from Cycle Computing describes how the company enabled HGST to spin up a 70,000-core cluster from AWS and then returned it 8 hours later.
Watch the video presentation: http://wp.me/p3RLHQ-dAG
MPI Requirements of the Network Layer
In this slidecast, Jeff Squyres from Cisco describes the proposed successor to the Linux verbs API that is designed to better serve the needs of MPI.
Learn more: http://blogs.cisco.com/performance/a-fun-thing-happened-on-the-way-to-the-openframeworks-discussion-today/#more-134672
Watch the video presentation: http://wp.me/p3RLHQ-beT
Deep Web - internetin karanlık Yuzu
internette ne kadar güvenliyiz, internetin yüzde kaçını kullanıyoruz, internette erişemediğimiz neler var, internette bir dakikada neler oluyor.
Viewers also liked (6)
DDN: Massively-Scalable Platforms and Solutions Engineered for the Big Data a...
DDN: Massively-Scalable Platforms and Solutions Engineered for the Big Data a...
AWS Customer Presentation - Cycle Computing - AWS Summit 2012 - NYC
AWS Customer Presentation - Cycle Computing - AWS Summit 2012 - NYC
Similar to Lustre Releases Update from LAD'14
Lustre Community Release Update
In this video from LAD'19 in Paris, Peter Jones from Whamcloud presents: Lustre Community Release Update.
"Lustre is a vibrant Open Source project with many organizations working on new features and improvements in parallel. We coordinate those efforts primarily through OpenSFS and EOFS. Meeting development target dates is a difficult task for any software project, but doubly so in a globally distributed Open Source project.
"Lustre is a massively parallel filesystem designed for high-performance, large-scale data. Lustre is mature and open source (GPLv2), running stably in production at many thousands of sites around the world. For a number of years, a majority of the world’s 100 fastest supercomputers have relied on Lustre for their storage needs. If you need lots of data fast and reliably, and value the flexibility of using a wide choice of block storage and want to become part of a world-wide open community, then Lustre is a good choice."
Watch the video: https://wp.me/p3RLHQ-lgN
Learn more: http://lustre.org
and
https://www.eofs.eu/events/lad19
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Whamcloud is Back: Lustre Today and Future
In this deck from the DDN User Group at ISC 2018, Robert Triendl from DDN presents: Lustre Today and Future.
On June 25, 2018, DDN announced that the company has acquired Intel’s Lustre File System business and related assets.
With their new Whamcloud Business Unit, DDN has already brought key Lustre developers on board to accelerate their high performance computing solutions.
"This important acquisition reinforces DDN’s presence as the global market leader for data at scale, while providing Lustre customers with enhanced field support and a well-funded technology roadmap. The acquisition also enables DDN to expand Lustre’s leading position from high performance computing (HPC) and Exascale into high growth markets such as analytics, AI and hybrid cloud."
Learn more: https://wp.me/p3RLHQ-iLM
and
http://ddn.com
Lustre Roadmap & Community Update
In this deck from DDN booth at SC18, Peter Jones from DDN presents: Lustre Roadmap & Community Update.
"With their new Whamcloud Business Unit, DDN has already brought key Lustre developers on board to accelerate their high performance computing solutions.
"This important acquisition reinforces DDN’s presence as the global market leader for data at scale, while providing Lustre customers with enhanced field support and a well-funded technology roadmap. The acquisition also enables DDN to expand Lustre’s leading position from high performance computing (HPC) and Exascale into high growth markets such as analytics, AI and hybrid cloud."
Learn more: http://ddn.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
The Path to Stackato
Stackato is a Platform as a Service built on Cloud Foundry that was first released in beta in 2011. Over time, Stackato added features like Linux containers, organization/user groups, buildpacks, an improved router, and integration with tools like Komodo. Stackato 3.0 will merge with the Cloud Foundry v2 codebase, include new timelines/dashboards and administrative views, improved cluster management, and auto-scaling support. The presentation discusses Stackato's relationship with Cloud Foundry and the importance of an open ecosystem approach.
The Path to Stackato
Stackato is a Platform as a Service built on Cloud Foundry that has gone through several versions since its initial beta release in 2011. Version 3.0 of Stackato will merge fully with Cloud Foundry version 2 and include improvements like new social features, updated administrative interfaces, improved cluster management, auto-scaling support, and a migration path from Stackato 2. It is based on the latest Cloud Foundry code and ActiveState is committed to participating in and strengthening the open source Cloud Foundry ecosystem.
2010-01-28 NSA Open Source User Group Meeting, Current & Future Linux on Syst...
RHEL 5.4 focused on virtualization improvements like full support for KVM hypervisor on x86_64, network performance enhancements through GRO, and storage updates including Ext4 bug fixes and a technology preview of XFS file system support. For System z, RHEL 5.4 included features like support for large volumes, FCP performance monitoring, shutdown action tools, and improved installation workflow.
State of the Dolphin, at db tech showcase Osaka 2014
- MySQL 5.7 provides significant performance improvements over previous versions through optimizations to InnoDB, the query optimizer, and other components.
- It includes many new features like improved replication, security enhancements, and new monitoring and management tools.
- Oracle continues to invest in driving MySQL innovation through both MySQL 5.7 development milestones and contributions from the MySQL community and users.
Whamcloud - Lustre for HPC and Ai
In this video from DDN User Group at SC18, Robert Triendl describes how the company is building tomorrow's Lustre file system technology for Exascale.
At SC18 in Dallas, DDN announced that its Whamcloud division is delivering professional support for Lustre clients on Arm architectures. With this support offering, organizations can confidently use Lustre in production environments, introduce new clients into existing Lustre infrastructures, and deploy Arm-based clusters of any size within test, development or production environments. As the use of Lustre continues to expand across HPC, artificial intelligence and data-intensive, performance-driven applications, the deployment of alternative architectures is on the rise.
Learn more: http://ddn.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
RHEL roadmap
Red Hat Enterprise Linux roadmap highlights included:
- RHEL 5 focusing on stability and basic hardware enablement, transitioning to production phase 2; RHEL 6 in production phase 1 with feature innovation and maintenance; RHEL 7 planned for 2013 with datacenter efficiency and virtualization/cloud enhancements.
- Upcoming KVM virtualization features focused on scalability, performance, and reliability like live migration and snapshots.
- Kernel developments emphasized optimizations for virtual memory, scheduling, resource management, networking and debugging tools.
- Hardware enablement discussed ongoing and planned support for Intel and ARM platforms, as well as power management and InfiniBand improvements.
2009-09-24 Get the Hype on System z Webinar with IBM, Current & Future Linux ...
Joint webinar series with Hans Picht (Linux on System z Lead, IBM). Covered recent release of Red Hat Enterprise Linux 5.4, which had the inclusion of Named Saved Segments (NSS), updated fiber channel, and rebasing of s390utils. Stepped through roadmap for RHEL on System z and gave update on CMM2 development activities.
2009-10-07 IBM zExpo 2009, Current & Future Linux on System z
This document summarizes the current and future Linux on System z technology presented by Shawn Wells of Red Hat. For RHEL 5.4, it highlights updates like support for Named Saved Segments and fiber channel drivers. It also provides a tentative roadmap for RHEL 6 and discusses ongoing CMM development. For future technology, it will cover areas like storage, networking, usability, and crypto that IBM and Red Hat are working on for Linux on System z.
[B34] MySQL最新ロードマップ – MySQL 5.7とその先へ by Ryusuke Kajiyama
- The document discusses recent developments and enhancements to MySQL, including performance improvements in MySQL 5.7 such as faster query execution, improved InnoDB engine, and new security features.
- MySQL 5.7 provides up to 230% performance gains over previous versions through improvements to scalability, transaction processing, and query optimization.
- New features in MySQL 5.7 include InnoDB page compression, improved replication throughput, and a new SYS schema for simplified monitoring of server performance.
Apache hadoop 3.x state of the union and upgrade guidance - Strata 2019 NY
The document discusses Apache Hadoop 3.x updates and provides guidance for upgrading to Hadoop 3. It covers community updates, features in YARN, Submarine, HDFS, and Ozone. Release plans are outlined for Hadoop, Submarine, and upgrades from Hadoop 2 to 3. Express upgrades are recommended over rolling upgrades for the major version change. The session summarizes that Hadoop 3 is an eagerly awaited release with many successful production uses, and that now is a good time for those not yet upgraded.
OSMC 2013 | Distributed Monitoring and Cloud Scaling for Web Apps by Fernando...
Der erste Teil der Präsentation wird zeigen, wie wir Nagios und WebInject in eine verteilte AWS EC2 Monitoring Infrastruktur integriert haben. Dies haben wir mit Hilfe von Event Handlers umgesetzt, die zusätzliche Checks an verschiedenen Punkten Schwellenwerte ermitteln und daraufhin einen Status anzeigen. Außerdem rufen wir verschiedene APIs von anderen Monitoring Tools wie Gomez Networks und Pingdom ab und nutzen deren Feeds. Für Benachrichtigungen haben wir CallWithUs (für VoIP Anrufe) und Clickatell für SMS Alarmierungen mit eingebunden.
Der zweite Teil des Talks wird ausführlich beschreiben, wie Hosts aktiviert und deaktiviert werden, Services von Anlagen die automatisch mit Amazon EC2 erzeugt werden und in der Cloud Infrastruktur autoscalieren. Außerdem werde ich aufzeigen, weshalb Host Gruppen so wichtig sind sowie deren kundenspezifische Implementierung. Im Anschluss werden Sie sehen wie Nagvis in Nagios mittels mklive_status broker intergiert wurde, um viele tolle Abbildungen zu erstellen, vor allem automatisch. Dabei werde ich Ihnen zeigen, wie einfach es ist verschiedene Nagios Systeme in einer Abbildung zusammenzufassen, um ein ganzheitliches Bild unseres Systems zu bekommen.
Distributed Monitoring and Cloud Scaling
The document discusses a distributed monitoring and cloud scaling architecture for web applications. It presents a solution using open source tools like Nagios, MK Livestatus, RESTlos, and iwatch to monitor infrastructure across multiple cloud regions. Automatic scaling is enabled through scripts that add and remove hosts from Nagios via RESTlos as instances start and stop. Hostgroups are used to associate services with server types. Dashboards provide a centralized view of monitoring data from all regions.
Fsoss2011
FreeNAS 8 is an open source network attached storage solution based on FreeBSD and using the ZFS filesystem. It provides enterprise features like snapshots, replication, and integration with LDAP/Active Directory. Planned features include a plugin architecture and migration tools. The presentation demonstrated the configuration workflow and encouraged participation in the FreeNAS community.
Linux io introduction-fudcon-2015-with-demo-slides
Linux provide facilities to expose emulated LUNs to initiators using Linux-IO (LIO) scsi target implementation . LIO not only support exposing conventional block devices but also supports other storage interfaces like file or memory based LUNs. Also it supports multiple fabric interfaces - FC, FCoE, iscsi and many more.
LIO can be used in SAN environments with minimal storage resources.
Native support for LIO in linux hypervisors and in Openstack make it a good storage option for cloud deployments.
This presentation includes demo slides with LIO iscsi target implementation.
Install Redis on Oracle Linux
A quick guide on installing Redis on Oracle Linux. This guide provides an insight on how to setup Redis to enable developers to quickly get started with the Open Source Redis solution on Oracle Linux based system
Devoxx Belgium 2015
The document summarizes key topics from the DEVOXX BE 2015 conference including Java 9 modular programming, HTTP/2 and Java 9, JSON API upgrades for Java EE 8, Spring roadmap, understanding Git internals, principles of microservices, best practices for Java deployment, and visualizing architecture. It provides an overview of 10 top talks at the conference focused on Java and web development topics.
What's New in PostgreSQL 9.6
In September 2016, the PostgreSQL community is rolling out PostgreSQL 9.6 which includes improvements in parallelism for query performance, overall performance improvements and the integration of foreign data sources.
This presentation introduces the new features of 9.6 and how they will benefit you.
- Parallel sequential scans, joins and aggregates
- Elimination of repetitive scanning of old data by autovacuum
- Synchronous replication now allows multiple standby servers for increased reliability
- Full-text search for phrases
- Support for remote joins, sorts, and updates in postgres_fdw
- Substantial performance improvements, especially in the area of improving scalability on many-CPU servers
If you have any questions on how to get started with Postgres, please email sales@enterprisedb.com
Similar to Lustre Releases Update from LAD'14 (20)
2010-01-28 NSA Open Source User Group Meeting, Current & Future Linux on Syst...
2010-01-28 NSA Open Source User Group Meeting, Current & Future Linux on Syst...
State of the Dolphin, at db tech showcase Osaka 2014
State of the Dolphin, at db tech showcase Osaka 2014
2009-09-24 Get the Hype on System z Webinar with IBM, Current & Future Linux ...
2009-09-24 Get the Hype on System z Webinar with IBM, Current & Future Linux ...
2009-10-07 IBM zExpo 2009, Current & Future Linux on System z
2009-10-07 IBM zExpo 2009, Current & Future Linux on System z
[B34] MySQL最新ロードマップ – MySQL 5.7とその先へ by Ryusuke Kajiyama
[B34] MySQL最新ロードマップ – MySQL 5.7とその先へ by Ryusuke Kajiyama
Apache hadoop 3.x state of the union and upgrade guidance - Strata 2019 NY
Apache hadoop 3.x state of the union and upgrade guidance - Strata 2019 NY
OSMC 2013 | Distributed Monitoring and Cloud Scaling for Web Apps by Fernando...
OSMC 2013 | Distributed Monitoring and Cloud Scaling for Web Apps by Fernando...
Linux io introduction-fudcon-2015-with-demo-slides
Linux io introduction-fudcon-2015-with-demo-slides
More from inside-BigData.com
Major Market Shifts in IT
The document discusses the top 5 technologies that all organizations must understand: digital transformation, quantum computing, IoT, 5G, and AI/HPC. It provides an overview of each technology including opportunities and threats to organizations. The document emphasizes that understanding these emerging technologies is mandatory as the information revolution changes many aspects of life and business.
Preparing to program Aurora at Exascale - Early experiences and future direct...
In this deck from IWOCL / SYCLcon 2020, Hal Finkel from Argonne National Laboratory presents: Preparing to program Aurora at Exascale - Early experiences and future directions.
"Argonne National Laboratory’s Leadership Computing Facility will be home to Aurora, our first exascale supercomputer. Aurora promises to take scientific computing to a whole new level, and scientists and engineers from many different fields will take advantage of Aurora’s unprecedented computational capabilities to push the boundaries of human knowledge. In addition, Aurora’s support for advanced machine-learning and big-data computations will enable scientific workflows incorporating these techniques along with traditional HPC algorithms. Programming the state-of-the-art hardware in Aurora will be accomplished using state-of-the-art programming models. Some of these models, such as OpenMP, are long-established in the HPC ecosystem. Other models, such as Intel’s oneAPI, based on SYCL, are relatively-new models constructed with the benefit of significant experience. Many applications will not use these models directly, but rather, will use C++ abstraction libraries such as Kokkos or RAJA. Python will also be a common entry point to high-performance capabilities. As we look toward the future, features in the C++ standard itself will become increasingly relevant for accessing the extreme parallelism of exascale platforms.
This presentation will summarize the experiences of our team as we prepare for Aurora, exploring how to port applications to Aurora’s architecture and programming models, and distilling the challenges and best practices we’ve developed to date. oneAPI/SYCL and OpenMP are both critical models in these efforts, and while the ecosystem for Aurora has yet to mature, we’ve already had a great deal of success. Importantly, we are not passive recipients of programming models developed by others. Our team works not only with vendor-provided compilers and tools, but also develops improved open-source LLVM-based technologies that feed both open-source and vendor-provided capabilities. In addition, we actively participate in the standardization of OpenMP, SYCL, and C++. To conclude, I’ll share our thoughts on how these models can best develop in the future to support exascale-class systems."
Watch the video: https://wp.me/p3RLHQ-lPT
Learn more: https://www.iwocl.org/iwocl-2020/conference-program/
and
https://www.anl.gov/topic/aurora
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Transforming Private 5G Networks
In this deck, Greg Wahl from Advantech presents: Transforming Private 5G Networks.
Advantech Networks & Communications Group is driving innovation in next-generation network solutions with their High Performance Servers. We provide business critical hardware to the world's leading telecom and networking equipment manufacturers with both standard and customized products. Our High Performance Servers are highly configurable platforms designed to balance the best in x86 server-class processing performance with maximum I/O and offload density. The systems are cost effective, highly available and optimized to meet next generation networking and media processing needs.
“Advantech’s Networks and Communication Group has been both an innovator and trusted enabling partner in the telecommunications and network security markets for over a decade, designing and manufacturing products for OEMs that accelerate their network platform evolution and time to market.” Said Advantech Vice President of Networks & Communications Group, Ween Niu. “In the new IP Infrastructure era, we will be expanding our expertise in Software Defined Networking (SDN) and Network Function Virtualization (NFV), two of the essential conduits to 5G infrastructure agility making networks easier to install, secure, automate and manage in a cloud-based infrastructure.”
In addition to innovation in air interface technologies and architecture extensions, 5G will also need a new generation of network computing platforms to run the emerging software defined infrastructure, one that provides greater topology flexibility, essential to deliver on the promises of high availability, high coverage, low latency and high bandwidth connections. This will open up new parallel industry opportunities through dedicated 5G network slices reserved for specific industries dedicated to video traffic, augmented reality, IoT, connected cars etc. 5G unlocks many new doors and one of the keys to its enablement lies in the elasticity and flexibility of the underlying infrastructure.
Advantech’s corporate vision is to enable an intelligent planet. The company is a global leader in the fields of IoT intelligent systems and embedded platforms. To embrace the trends of IoT, big data, and artificial intelligence, Advantech promotes IoT hardware and software solutions with the Edge Intelligence WISE-PaaS core to assist business partners and clients in connecting their industrial chains. Advantech is also working with business partners to co-create business ecosystems that accelerate the goal of industrial intelligence."
Watch the video: https://wp.me/p3RLHQ-lPQ
* Company website: https://www.advantech.com/
* Solution page: https://www2.advantech.com/nc/newsletter/NCG/SKY/benefits.html
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
In this deck from the Stanford HPC Conference, Katie Lewis from Lawrence Livermore National Laboratory presents: The Incorporation of Machine Learning into Scientific Simulations at Lawrence Livermore National Laboratory.
"Scientific simulations have driven computing at Lawrence Livermore National Laboratory (LLNL) for decades. During that time, we have seen significant changes in hardware, tools, and algorithms. Today, data science, including machine learning, is one of the fastest growing areas of computing, and LLNL is investing in hardware, applications, and algorithms in this space. While the use of simulations to focus and understand experiments is well accepted in our community, machine learning brings new challenges that need to be addressed. I will explore applications for machine learning in scientific simulations that are showing promising results and further investigation that is needed to better understand its usefulness."
Watch the video: https://youtu.be/NVwmvCWpZ6Y
Learn more: https://computing.llnl.gov/research-area/machine-learning
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
In this deck from the Stanford HPC Conference, DK Panda from Ohio State University presents: How to Achieve High-Performance, Scalable and Distributed DNN Training on Modern HPC Systems?
"This talk will start with an overview of challenges being faced by the AI community to achieve high-performance, scalable and distributed DNN training on Modern HPC systems with both scale-up and scale-out strategies. After that, the talk will focus on a range of solutions being carried out in my group to address these challenges. The solutions will include: 1) MPI-driven Deep Learning, 2) Co-designing Deep Learning Stacks with High-Performance MPI, 3) Out-of- core DNN training, and 4) Hybrid (Data and Model) parallelism. Case studies to accelerate DNN training with popular frameworks like TensorFlow, PyTorch, MXNet and Caffe on modern HPC systems will be presented."
Watch the video: https://youtu.be/LeUNoKZVuwQ
Learn more: http://web.cse.ohio-state.edu/~panda.2/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
In this deck from the Stanford HPC Conference, Nick Nystrom and Paola Buitrago provide an update from the Pittsburgh Supercomputing Center.
Nick Nystrom is Chief Scientist at the Pittsburgh Supercomputing Center (PSC). Nick is architect and PI for Bridges, PSC's flagship system that successfully pioneered the convergence of HPC, AI, and Big Data. He is also PI for the NIH Human Biomolecular Atlas Program’s HIVE Infrastructure Component and co-PI for projects that bring emerging AI technologies to research (Open Compass), apply machine learning to biomedical data for breast and lung cancer (Big Data for Better Health), and identify causal relationships in biomedical big data (the Center for Causal Discovery, an NIH Big Data to Knowledge Center of Excellence). His current research interests include hardware and software architecture, applications of machine learning to multimodal data (particularly for the life sciences) and to enhance simulation, and graph analytics.
Watch the video: https://youtu.be/LWEU1L1o7yY
Learn more: https://www.psc.edu/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
HPC Impact: EDA Telemetry Neural Networks
The document discusses using systems intelligence and artificial intelligence/neural networks to enhance semiconductor electronic design automation (EDA) workflows by collecting telemetry data from EDA jobs and infrastructure and analyzing it using complex event processing, machine learning models, and messaging substrates to provide insights that could optimize EDA pipelines and infrastructure. The approach aims to allow both internal and external augmentation of EDA processes and environments through unsupervised and incremental learning.
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
In this deck from the Stanford HPC Conference, Nicole Xu from Stanford University describes how she transformed a common jellyfish into a bionic creature that is part animal and part machine.
"Animal locomotion and bioinspiration have the potential to expand the performance capabilities of robots, but current implementations are limited. Mechanical soft robots leverage engineered materials and are highly controllable, but these biomimetic robots consume more power than corresponding animal counterparts. Biological soft robots from a bottom-up approach offer advantages such as speed and controllability but are limited to survival in cell media. Instead, biohybrid robots that comprise live animals and self- contained microelectronic systems leverage the animals’ own metabolism to reduce power constraints and body as an natural scaffold with damage tolerance. We demonstrate that by integrating onboard microelectronics into live jellyfish, we can enhance propulsion up to threefold, using only 10 mW of external power input to the microelectronics and at only a twofold increase in cost of transport to the animal. This robotic system uses 10 to 1000 times less external power per mass than existing swimming robots in literature and can be used in future applications for ocean monitoring to track environmental changes."
Watch the video: https://youtu.be/HrmJFyvInj8
Learn more: https://sanfrancisco.cbslocal.com/2020/02/05/stanford-research-project-common-jellyfish-bionic-sea-creatures/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Machine Learning for Weather Forecasts
In this deck from the Stanford HPC Conference, Peter Dueben from the European Centre for Medium-Range Weather Forecasts (ECMWF) presents: Machine Learning for Weather Forecasts.
"I will present recent studies that use deep learning to learn the equations of motion of the atmosphere, to emulate model components of weather forecast models and to enhance usability of weather forecasts. I will than talk about the main challenges for the application of deep learning in cutting-edge weather forecasts and suggest approaches to improve usability in the future."
Peter is contributing to the development and optimization of weather and climate models for modern supercomputers. He is focusing on a better understanding of model error and model uncertainty, on the use of reduced numerical precision that is optimised for a given level of model error, on global cloud- resolving simulations with ECMWF's forecast model, and the use of machine learning, and in particular deep learning, to improve the workflow and predictions. Peter has graduated in Physics and wrote his PhD thesis at the Max Planck Institute for Meteorology in Germany. He worked as Postdoc with Tim Palmer at the University of Oxford and has taken up a position as University Research Fellow of the Royal Society at the European Centre for Medium-Range Weather Forecasts (ECMWF) in 2017.
Watch the video: https://youtu.be/ks3fkRj8Iqc
Learn more: https://www.ecmwf.int/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
HPC AI Advisory Council Update
In this deck, Gilad Shainer from the HPC AI Advisory Council describes how this organization fosters innovation in the high performance computing community.
"The HPC-AI Advisory Council’s mission is to bridge the gap between high-performance computing (HPC) and Artificial Intelligence (AI) use and its potential, bring the beneficial capabilities of HPC and AI to new users for better research, education, innovation and product manufacturing, bring users the expertise needed to operate HPC and AI systems, provide application designers with the tools needed to enable parallel computing, and to strengthen the qualification and integration of HPC and AI system products."
Watch the video: https://wp.me/p3RLHQ-lNz
Learn more: http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Fugaku Supercomputer joins fight against COVID-19
Today RIKEN in Japan announced that the Fugaku supercomputer will be made available for research projects aimed to combat COVID-19.
"Fugaku is currently being installed and is scheduled to be available to the public in 2021. However, faced with the devastating disaster unfolding before our eyes, RIKEN and MEXT decided to make a portion of the computational resources of Fugaku available for COVID-19-related projects ahead of schedule while continuing the installation process.
Fugaku is being developed not only for the progress in science, but also to help build the society dubbed as the “Society 5.0” by the Japanese government, where all people will live safe and comfortable lives. The current initiative to fight against the novel coronavirus is driven by the philosophy behind the development of Fugaku."
Initial Projects
Exploring new drug candidates for COVID-19 by "Fugaku"
Yasushi Okuno, RIKEN / Kyoto University
Prediction of conformational dynamics of proteins on the surface of SARS-Cov-2 using Fugaku
Yuji Sugita, RIKEN
Simulation analysis of pandemic phenomena
Nobuyasu Ito, RIKEN
Fragment molecular orbital calculations for COVID-19 proteins
Yuji Mochizuki, Rikkyo University
Energy Efficient Computing using Dynamic Tuning
In this deck from the Performance Optimisation and Productivity group, Lubomir Riha from IT4Innovations presents: Energy Efficient Computing using Dynamic Tuning.
"We now live in a world of power-constrained architectures and systems and power consumption represents a significant cost factor in the overall HPC system economy. For these reasons, in recent years researchers, supercomputing centers and major vendors have developed new tools and methodologies to measure and optimize the energy consumption of large-scale high performance system installations. Due to the link between energy consumption, power consumption and execution time of an application executed by the final user, it is important for these tools and the methodology used to consider all these aspects, empowering the final user and the system administrator with the capability of finding the best configuration given different high level objectives.
This webinar focused on tools designed to improve the energy-efficiency of HPC applications using a methodology of dynamic tuning of HPC applications, developed under the H2020 READEX project. The READEX methodology has been designed for exploiting the dynamic behaviour of software. At design time, different runtime situations (RTS) are detected and optimized system configurations are determined. RTSs with the same configuration are grouped into scenarios, forming the tuning model. At runtime, the tuning model is used to switch system configurations dynamically.
The MERIC tool, that implements the READEX methodology, is presented. It supports manual or binary instrumentation of the analysed applications to simplify the analysis. This instrumentation is used to identify and annotate the significant regions in the HPC application. Automatic binary instrumentation annotates regions with significant runtime. Manual instrumentation, which can be combined with automatic, allows code developer to annotate regions of particular interest."
Watch the video: https://wp.me/p3RLHQ-lJP
Learn more: https://pop-coe.eu/blog/14th-pop-webinar-energy-efficient-computing-using-dynamic-tuning
and
https://code.it4i.cz/vys0053/meric
Sign up for our insideHPC Newsletter: http://insidehpc.com/newslett
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
The document discusses how DDN A3I storage solutions and Nvidia's SuperPOD platform can enable HPC at scale. It provides details on DDN's A3I appliances that are optimized for AI and deep learning workloads and validated for Nvidia's DGX-2 SuperPOD reference architecture. The solutions are said to deliver the fastest performance, effortless scaling, reliability and flexibility for data-intensive workloads.
State of ARM-based HPC
In this deck, Paul Isaacs from Linaro presents: State of ARM-based HPC. This talk provides an overview of applications and infrastructure services successfully ported to Aarch64 and benefiting from scale.
"With its debut on the TOP500, the 125,000-core Astra supercomputer at New Mexico’s Sandia Labs uses Cavium ThunderX2 chips to mark Arm’s entry into the petascale world. In Japan, the Fujitsu A64FX Arm-based CPU in the pending Fugaku supercomputer has been optimized to achieve high-level, real-world application performance, anticipating up to one hundred times the application execution performance of the K computer. K was the first computer to top 10 petaflops in 2011."
Watch the video: https://wp.me/p3RLHQ-lIT
Learn more: https://www.linaro.org/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Versal Premium ACAP for Network and Cloud Acceleration
Today Xilinx announced Versal Premium, the third series in the Versal ACAP portfolio. The Versal Premium series features highly integrated, networked and power-optimized cores and the industry’s highest bandwidth and compute density on an adaptable platform. Versal Premium is designed for the highest bandwidth networks operating in thermally and spatially constrained environments, as well as for cloud providers who need scalable, adaptable application acceleration.
Versal is the industry’s first adaptive compute acceleration platform (ACAP), a revolutionary new category of heterogeneous compute devices with capabilities that far exceed those of conventional silicon architectures. Developed on TSMC’s 7-nanometer process technology, Versal Premium combines software programmability with dynamically configurable hardware acceleration and pre-engineered connectivity and security features to enable a faster time-to- market. The Versal Premium series delivers up to 3X higher throughput compared to current generation FPGAs, with built-in Ethernet, Interlaken, and cryptographic engines that enable fast and secure networks. The series doubles the compute density of currently deployed mainstream FPGAs and provides the adaptability to keep pace with increasingly diverse and evolving cloud and networking workloads.
Learn more: https://insidehpc.com/2020/03/xilinx-announces-versal-premium-acap-for-network-and-cloud-acceleration/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
In this video from the Rice Oil & Gas Conference, Chin Fang from Zettar presents: Moving Massive Amounts of Data across Any Distance Efficiently.
The objective of this talk is to present two on-going projects aiming at improving and ensuring highly efficient bulk transferring or streaming of massive amounts of data over digital connections across any distance. It examines the current state of the art, a few very common misconceptions, the differences among the three major type of data movement solutions, a current initiative attempting to improve the data movement efficiency from the ground up, and another multi-stage project that shows how to conduct long distance large scale data movement at speed and scale internationally. Both projects have real world motivations, e.g. the ambitious data transfer requirements of Linac Coherent Light Source II (LCLS-II) [1], a premier preparation project of the U.S. DOE Exascale Computing Initiative (ECI) [2]. Their immediate goals are described and explained, together with the solution used for each. Findings and early results are reported. Possible future works are outlined.
Watch the video: https://wp.me/p3RLHQ-lBX
Learn more: https://www.zettar.com/
and
https://rice2020oghpc.rice.edu/program-2/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Scaling TCO in a Post Moore's Era
In this deck from the Rice Oil & Gas Conference, Bradley McCredie from AMD presents: Scaling TCO in a Post Moore's Law Era.
"While foundries bravely drive forward to overcome the technical and economic challenges posed by scaling to 5nm and beyond, Moore’s law alone can provide only a fraction of the performance / watt and performance / dollar gains needed to satisfy the demands of today’s high performance computing and artificial intelligence applications. To close the gap, multiple strategies are required. First, new levels of innovation and design efficiency will supplement technology gains to continue to deliver meaningful improvements in SoC performance. Second, heterogenous compute architectures will create x-factor increases of performance efficiency for the most critical applications. Finally, open software frameworks, APIs, and toolsets will enable broad ecosystems of application level innovation."
Watch the video:
Learn more: http://amd.com
and
https://rice2020oghpc.rice.edu/program-2/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
CUDA-Python and RAPIDS for blazing fast scientific computing
In this deck from the ECSS Symposium, Abe Stern from NVIDIA presents: CUDA-Python and RAPIDS for blazing fast scientific computing.
"We will introduce Numba and RAPIDS for GPU programming in Python. Numba allows us to write just-in-time compiled CUDA code in Python, giving us easy access to the power of GPUs from a powerful high-level language. RAPIDS is a suite of tools with a Python interface for machine learning and dataframe operations. Together, Numba and RAPIDS represent a potent set of tools for rapid prototyping, development, and analysis for scientific computing. We will cover the basics of each library and go over simple examples to get users started. Finally, we will briefly highlight several other relevant libraries for GPU programming."
Watch the video: https://wp.me/p3RLHQ-lvu
Learn more: https://developer.nvidia.com/rapids
and
https://www.xsede.org/for-users/ecss/ecss-symposium
Sign up for our insideHPC Newsletter: http://insidehp.com/newsletter
Introducing HPC with a Raspberry Pi Cluster
In this deck from FOSDEM 2020, Colin Sauze from Aberystwyth University describes the development of a RaspberryPi cluster for teaching an introduction to HPC.
"The motivation for this was to overcome four key problems faced by new HPC users:
* The availability of a real HPC system and the effect running training courses can have on the real system, conversely the availability of spare resources on the real system can cause problems for the training course.
* A fear of using a large and expensive HPC system for the first time and worries that doing something wrong might damage the system.
* That HPC systems are very abstract systems sitting in data centres that users never see, it is difficult for them to understand exactly what it is they are using.
* That new users fail to understand resource limitations, in part because of the vast resources in modern HPC systems a lot of mistakes can be made before running out of resources. A more resource constrained system makes it easier to understand this.
The talk will also discuss some of the technical challenges in deploying an HPC environment to a Raspberry Pi and attempts to keep that environment as close to a "real" HPC as possible. The issue to trying to automate the installation process will also be covered."
Learn more: https://github.com/colinsauze/pi_cluster
and
https://fosdem.org/2020/schedule/events/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Overview of HPC Interconnects
In this deck from ATPESC 2019, Ken Raffenetti from Argonne presents an overview of HPC interconnects.
"The Argonne Training Program on Extreme-Scale Computing (ATPESC) provides intensive, two-week training on the key skills, approaches, and tools to design, implement, and execute computational science and engineering applications on current high-end computing systems and the leadership-class computing systems of the future."
Watch the video: https://wp.me/p3RLHQ-luc
Learn more: https://extremecomputingtraining.anl.gov/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
More from inside-BigData.com (20)
Preparing to program Aurora at Exascale - Early experiences and future direct...
Preparing to program Aurora at Exascale - Early experiences and future direct...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
Versal Premium ACAP for Network and Cloud Acceleration
Versal Premium ACAP for Network and Cloud Acceleration
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
CUDA-Python and RAPIDS for blazing fast scientific computing
CUDA-Python and RAPIDS for blazing fast scientific computing
Recently uploaded
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks creates AI-boosted business software that helps employees work more efficiently and effectively. Managing data across multiple RDBMS and NoSQL databases was already a challenge at their current scale. To prepare for 10X growth, they knew it was time to rethink their database strategy. Learn how they architected a solution that would simplify scaling while keeping costs under control.
Introduction of Cybersecurity with OSS at Code Europe 2024
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Direct losses from downtime in 1 minute = $5-$10 thousand dollars. Reputation is priceless.
As part of the talk, we will consider the architectural strategies necessary for the development of highly loaded fintech solutions. We will focus on using queues and streaming to efficiently work and manage large amounts of data in real-time and to minimize latency.
We will focus special attention on the architectural patterns used in the design of the fintech system, microservices and event-driven architecture, which ensure scalability, fault tolerance, and consistency of the entire system.
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Pitangent Analytics & Technology Solutions Pvt. Ltd
Discover top-tier mobile app development services, offering innovative solutions for iOS and Android. Enhance your business with custom, user-friendly mobile applications.Main news related to the CCS TSI 2023 (2023/1695)
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Session 1 - Intro to Robotic Process Automation.pdf
👉 Check out our full 'Africa Series - Automation Student Developers (EN)' page to register for the full program:
https://bit.ly/Automation_Student_Kickstart
In this session, we shall introduce you to the world of automation, the UiPath Platform, and guide you on how to install and setup UiPath Studio on your Windows PC.
📕 Detailed agenda:
What is RPA? Benefits of RPA?
RPA Applications
The UiPath End-to-End Automation Platform
UiPath Studio CE Installation and Setup
💻 Extra training through UiPath Academy:
Introduction to Automation
UiPath Business Automation Platform
Explore automation development with UiPath Studio
👉 Register here for our upcoming Session 2 on June 20: Introduction to UiPath Studio Fundamentals: https://community.uipath.com/events/details/uipath-lagos-presents-session-2-introduction-to-uipath-studio-fundamentals/
"Choosing proper type of scaling", Olena Syrota
Imagine an IoT processing system that is already quite mature and production-ready and for which client coverage is growing and scaling and performance aspects are life and death questions. The system has Redis, MongoDB, and stream processing based on ksqldb. In this talk, firstly, we will analyze scaling approaches and then select the proper ones for our system.
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
This LF Energy webinar took place June 20, 2024. It featured:
-Alex Thornton, LF Energy
-Hallie Cramer, Google
-Daniel Roesler, UtilityAPI
-Henry Richardson, WattTime
In response to the urgency and scale required to effectively address climate change, open source solutions offer significant potential for driving innovation and progress. Currently, there is a growing demand for standardization and interoperability in energy data and modeling. Open source standards and specifications within the energy sector can also alleviate challenges associated with data fragmentation, transparency, and accessibility. At the same time, it is crucial to consider privacy and security concerns throughout the development of open source platforms.
This webinar will delve into the motivations behind establishing LF Energy’s Carbon Data Specification Consortium. It will provide an overview of the draft specifications and the ongoing progress made by the respective working groups.
Three primary specifications will be discussed:
-Discovery and client registration, emphasizing transparent processes and secure and private access
-Customer data, centering around customer tariffs, bills, energy usage, and full consumption disclosure
-Power systems data, focusing on grid data, inclusive of transmission and distribution networks, generation, intergrid power flows, and market settlement data
AppSec PNW: Android and iOS Application Security with MobSF
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
Essentials of Automations: Exploring Attributes & Automation Parameters
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
How we managed to grow and scale a RAG application from zero to thousands of users in 7 months. Lessons from technical challenges around managing high load for LLMs, RAGs and Vector databases.
Apps Break Data
How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
What began over 115 years ago as a supplier of precision gauges to the automotive industry has evolved into being an industry leader in the manufacture of product branding, automotive cockpit trim and decorative appliance trim. Value-added services include in-house Design, Engineering, Program Management, Test Lab and Tool Shops.
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Northern Engraving | Nameplate Manufacturing Process - 2024
Manufacturing custom quality metal nameplates and badges involves several standard operations. Processes include sheet prep, lithography, screening, coating, punch press and inspection. All decoration is completed in the flat sheet with adhesive and tooling operations following. The possibilities for creating unique durable nameplates are endless. How will you create your brand identity? We can help!
Demystifying Knowledge Management through Storytelling
The Department of Veteran Affairs (VA) invited Taylor Paschal, Knowledge & Information Management Consultant at Enterprise Knowledge, to speak at a Knowledge Management Lunch and Learn hosted on June 12, 2024. All Office of Administration staff were invited to attend and received professional development credit for participating in the voluntary event.
The objectives of the Lunch and Learn presentation were to:
- Review what KM ‘is’ and ‘isn’t’
- Understand the value of KM and the benefits of engaging
- Define and reflect on your “what’s in it for me?”
- Share actionable ways you can participate in Knowledge - - Capture & Transfer
Recently uploaded (20)
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Freshworks Rethinks NoSQL for Rapid Scaling & Cost-Efficiency
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Nameplate Manufacturing Process - 2024
Northern Engraving | Nameplate Manufacturing Process - 2024
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
Lustre Releases Update from LAD'14
- 1. Lustre* Releases Peter Jones Lustre Support and Releases Manager *Other names and brands may be claimed as the property of others.
- 2. Lustre 2.4.x has been most common choice for new deployments Recent shift towards Lustre 2.5.x 2 Lustre* Release Trends Lustre Versions in Production *Other names and brands may be claimed as the property of others. 40 35 30 25 20 15 10 5 0 1.6.x 1.8.x 2.0 2.1.x 2.2 2.3 2.4.x 2.5.x Source: OpenSFS Survey March 2014 76 Respondents could make multiple selections
- 3. Lustre 2.1.0 declared GA Oct 2011 RHEL 6.x servers and large LUNs the main attraction Still a large number of sites in production on 2.1.x but many of larger sites have now upgraded § NASA/CEA/LLNL all upgraded to more current releases Formerly maintenance release stream § Latest release 2.1.6 June 2013 3 Lustre* 2.1.x *Other names and brands may be claimed as the property of others.
- 4. Lustre 2.4.0 declared GA May 2013 Features include DNE Remote Directories (LU-1187); Network Request Scheduler (LU-398) and ZFS support (LU-1305) Most active codeline over past year § NASA/CEA/LLNL/ORNL all running in production § DDN, Bull and others using for deployments Formerly maintenance release stream § Latest release 2.4.3 Mar 2014 4 Lustre* 2.4.x *Other names and brands may be claimed as the property of others.
- 5. Lustre 2.5.0 declared GA Oct 2013 HSM (LU-3608) is the primary feature § Manages data transfer between different storage types Indications are that this codeline will be widely adopted § Many upgrades underway Present maintenance release stream § Latest release 2.5.3 Sept 2014 § Lustre 2.5.4 targeted for Q4 2014 5 Lustre* 2.5.x *Other names and brands may be claimed as the property of others.
- 6. Declared GA July 2014 Several new features § LFSCK MDT-OST Consistency (LU-1267) § Single client IO performance (LU-3321) § DNE Striped directories (LU-3531) preview Much groundwork to support newer kernels Feature release only; no maintenance releases planned 6 Lustre* 2.6 *Other names and brands may be claimed as the property of others.
- 7. Commits between 2.5.50 and 2.6 GA by Organization 7 Lustre* 2.6 – Code Contributions ANU 1 Bull 11 CEA 7 Cray 21 DDN 33 EMC 9 GSI 6 Intel 654 Xyratex 31 Suse 7 LLNL 22 ORNL 83 Suse 122 Xyratex SGI 70 *Other names and brands may be claimed as the property of others. IU 6 SGI 3 ANU 4 Bull 952 CEA 611 Cray 6389 DDN 3021 EMC 2840 GSI 125 Intel 80179 IU 5679 LLNL 1324 ORNL 28609 1945 Number of Commits Lines of Code changed
- 8. Version Commits LOC Developers Organizations 1.8.0 997 291K 41 1 2.1.0 752 92K 55 7 2.2.0 329 58K 42 10 2.3.0 586 87K 52 13 2.4.0 1123 348K 69 19 2.5.0 471 102K 70 15 2.6.0 894 132K 76 14 8 Lustre* Version Statistics *Other names and brands may be claimed as the property of others.
- 9. Targeted GA Feb 2015 § Feature freeze Oct 31st 2014 Several new features targeted for this release § UID Mapping (LU-3527) § LFSCK MDT-MDT Consistency (LU-4788) § Dynamic LNET Configuration (LU-2456) Will add RHEL7 client support § Likely SLES12 client support too when GA confirmed Interop and upgrades supported with 2.6 and 2.5.x releases Feature release only; no maintenance releases planned 9 Lustre* 2.7 *Other names and brands may be claimed as the property of others.
- 10. First appeared in staging area in 3.11 kernel Client is slightly ahead of a 2.4.0 client in functionality Some sites report on mailing lists to be running in production Linux distribution releases now contain in-kernel Lustre client § Ubuntu 14.04 and SLES12 do; RHEL 7 just missed out (3.10) § Poses some logistical challenges (LU-5628) Working with upstream community to get Lustre out of staging § Remove typedefs (LU-5478) § Deprecate proc/fs/lustre (LU-5030) § Aiming to complete much of this work for Lustre 2.7 10 Upstream Lustre* Client *Other names and brands may be claimed as the property of others.
- 11. Well-established release validation practices § Automated functional regression tests across test matrix § SWL runs on Hyperion § Execution of feature test plans Continuing to evolve testing practices § Fault injection § Aged file system testing § Soak testing § Static code analysis tools 11 Lustre* Release Testing *Other names and brands may be claimed as the property of others.
- 12. Latest version of user documentation dynamically available to download § http://lustre.opensfs.org/documentation/ See Richard Henwood’s recent LUG presentation for details on how to contribute § http://cdn.opensfs.org/wp-content/uploads/2013/05/ Henwood_manual_LUG13_FINAL_v2.pdf If you know of gaps then please open an LUDOC ticket § If you have not got time to work out the correct format to submit then unformatted text will provide a starting point for someone else to complete Internals documention also being improved (LU-1892) 12 Lustre* Release Documentation *Other names and brands may be claimed as the property of others.
- 13. Combines previous TWG and CDWG § Chris Morrone of LLNL is lead Single forum for all Lustre development matters § Oversees entire Lustre development cycle § Maintains the roadmap § Plans major releases § Collects requirements for future Lustre features § Sets priorities for test matrix For more information visit the wiki http://wiki.opensfs.org/Lustre_Working_Group 13 OpenSFS Lustre* Working Group *Other names and brands may be claimed as the property of others.