Alluxio on AWS EMR Fast Storage Access & Sharing for Spark

•

2 likes•842 views

Meetup.com Sr Data Platform Eng Chengzhi Zhao presents on Alluxio on AWS EMR Fast Storage Access & Sharing for Spark at the Alluxio open source New York meetup hosted by Meetup on July 10. Check out more upcoming events: https://www.alluxio.io/events/

Alluxio Open Source New York Meetup
Alluxio on AWS EMR
Fast storage access and sharing for Spark

Table of Content
● Spark and Alluxio
● AWS EMR and deployment of Alluxio
● Demo

About me
Chengzhi Zhao
● Data Platform Engineer at Meetup 2.5 years, 4 years in
retail and healthcare
● Area of Interest: Build scalable, reliable and
maintainable data infrastructure and tools

How do we run spark jobs?
EMR Dataproc HDInsight on
premise
S3
GCS
DB
HDFS
Compute Storage

To Summarize...
● Multiple cloud providers
● Storage layer is separated from compute
layer
● Data skew/Data Format
● Failure/Restart
● Data sharing is diﬃcult

DATA LOCALITY
Spark tries to execute tasks as close to the data as possible to minimize
data transfer (over the wire).
Preferred Locality Order: PROCESS_LOCAL, NODE_LOCAL,
RACK_LOCAL, or ANY

Spark Read & Write
spark.read.parquet("alluxio://")
val rdd = sc.textFile("alluxio://")
df.spark.write.parquet("alluxio://")
rdd.saveAsTextFile("alluxio://")

Caching
● Avoid duplicated cache
● Crash/Restart

Tiered Storage
By default, Alluxio only enables a single, memory tier.
● MEM (Memory)
● SSD (Solid State Drives)
● HDD (Hard Disk Drives)

Tiered Storage Conﬁgurations
alluxio.worker.tieredstore.levels=2
alluxio.worker.tieredstore.level0.alias=MEM
alluxio.worker.tieredstore.level0.dirs.quota=24GB
alluxio.worker.tieredstore.level0.watermark.high.ratio=0.9
alluxio.worker.tieredstore.level0.watermark.low.ratio=0.7
alluxio.worker.tieredstore.level1.alias=HDD
alluxio.worker.tieredstore.level1.dirs.quota=300GB

Preload data into Alluxio Uniformly
$ bin/alluxio fs load /path/to/load
Data
Warehouse

EMR (Elastic MapReduce)
Amazon EMR simplifies building and operating big data environments
and applications. EMR features include easy provisioning, scaling, and
reconfiguring of clusters, and notebooks for collaborative development.
Applications: Flink, Ganglia, Hadoop, HBase, HCatalog, Hive, Hue,
JupyterHub, Livy, Mahout, MXNet, Oozie, Phoenix, Pig, Presto, Spark,
Sqoop, TensorFlow, Tez, Zeppelin, and ZooKeeper.

$Setup Alluxio on EMR aws emr create-cluster --applications Name=Spark --release-label emr-5.23.0 --configurations ... --bootstrap-actions '[{"Path":"s3://test/bootstrap_cz.sh"]' --region us-east-1 core-site "fs.alluxio.impl": "alluxio.hadoop.FileSystem" "fs.AbstractFileSystem.alluxio.impl": "alluxio.hadoop.AlluxioFileSystem" spark-defaults "spark.driver.extraClassPath": ":/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrf s/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/sh are/aws/emr/security/lib/*:/opt/alluxio-1.8.1-hadoop-2.8/client/alluxio-1.8.1-client.jar", "spark.executor.extraClassPath": ":/opt/alluxio-1.8.1-hadoop-2.8/client/alluxio-1.8.1-client.jar"$

$Bootstrap ismaster=`cat /mnt/var/lib/info/instance.json | jq -r '.isMaster'` masterdns=`cat /mnt/var/lib/info/job-flow.json | jq -r '.masterPrivateDnsName'` …. if [[ ${ismaster} == "true" ]]; then sudo ./bin/alluxio bootstrapConf ${masterdns} sudo ./bin/alluxio format sudo ./bin/alluxio-start.sh master else sudo ./bin/alluxio bootstrapConf ${masterdns} initialize_alluxio sudo ./bin/alluxio format sudo ./bin/alluxio-start.sh worker Mount fi ….$

Hardware
● r5.2xlarge
● 8 vCore
● 64 GiB memory
● 3 instances

Q & A
https://github.com/ChengzhiZhao/Alluxio-EMR-bootstrap

ApacheCon 2021 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Lu Qiu Bin Fan Alluxio’s capabilities as a Data Orchestration framework have encouraged users to onboard more of their data-driven applications to an Alluxio powered data access layer. Driven by strong interests from our open-source community, the core team of Alluxio started to re-design an efficient and transparent way for users to leverage data orchestration through the POSIX interface. This effort has a lot of progress with the collaboration with engineers from Microsoft, Alibaba and Tencent. Particularly, we have introduced a new JNI-based FUSE implementation to support POSIX data access, created a more efficient way to integrate Alluxio with FUSE service, as well as many improvements in relevant data operations like more efficient distributedLoad, optimizations on listing or calculating directories with a massive amount of files, which are common in model training. We will also share our engineering lessons and roadmap in future releases to support Machine Learning applications.

Simplified Data Preparation for Machine Learning in Hybrid and Multi Clouds

Alluxio, Inc.

Fast Big Data Analytics with Spark on Tachyon

Alluxio, Inc.

This document discusses using Tachyon as a cache layer to accelerate big data analytics workloads at Baidu. It describes how Tachyon provides a 30x speedup over directly accessing remote storage by serving data from an in-memory cache. The document outlines Baidu's architecture using Tachyon with Spark SQL, discusses problems encountered in production and their solutions, and proposes advanced features like tiered storage to expand Tachyon's caching capacity.

A Reliable Memory-Centric Distributed Storage System

Alluxio, Inc.

This document provides an overview of Tachyon, an open-source memory-centric distributed storage system. It discusses Tachyon's motivation of leveraging faster memory storage, architecture that manages data across tiers of storage, and status as an Apache project used in production at over 50 companies. The document outlines Tachyon's ability to improve data sharing and fault tolerance across different compute frameworks. It shares several customer use cases and previews new features around lineage and tiered storage. The overall goal of Tachyon is to better assist other components with distributed memory-centric storage.

Speeding Up Spark Performance using Alluxio at China Unicom

Alluxio, Inc.

Cybersecurity and fraud detection at ING Bank using Presto & Alluxio on S3

Alluxio, Inc.

This document summarizes a presentation on cybersecurity detection at ING Bank. It discusses ING's distributed environment with data sources all over the world, the challenges of this for security, and how they use a SIEM system for detection. However, SIEMs have limitations like being mostly rule-based and slow. The presentation then demonstrates Alluxio as a solution for faster data access and detection. It shows how Alluxio can check for obfuscated powershell scripts in just 2 seconds using cosine similarity, whereas detection in a SIEM would be slower. Live demos are also provided of obfuscating a script and detecting the obfuscation.

Securely Enhancing Data Access in Hybrid Cloud with Alluxio

Alluxio, Inc.

Alluxio Global Online Meetup Apr 23, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Jiao (Jennie) Wang, Intel Tsai Louie, Intel Bin Fan, Alluxio Today, many people run deep learning applications with training data from separate storage such as object storage or remote data centers. This presentation will demo the Intel Analytics Zoo + Alluxio stack, an architecture that enables high performance while keeping cost and resource efficiency balanced without network being I/O bottlenecked. Intel Analytics Zoo is a unified data analytics and AI platform open-sourced by Intel. It seamlessly unites TensorFlow, Keras, PyTorch, Spark, Flink, and Ray programs into an integrated pipeline, which can transparently scale from a laptop to large clusters to process production big data. Alluxio, as an open-source data orchestration layer, accelerates data loading and processing in Analytics Zoo deep learning applications. This talk, we will go over: - What is Analytics Zoo and how it works - How to run Analytics Zoo with Alluxio in deep learning applications - Initial performance benchmark results using the Analytics Zoo + Alluxio stack

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Alluxio Webinar September 22, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Hybrid data lake on google cloud with alluxio and dataproc

Alluxio, Inc.

Accelerating Hive with Alluxio on S3

Alluxio, Inc.

This document summarizes an Alluxio office hour on accelerating Hive with Alluxio on AWS S3. It discusses why S3 is popular but has performance limitations for analytics. Alluxio provides data orchestration to add caching and improve performance. The office hour demonstrated creating an EMR cluster with Alluxio using bootstrap actions, querying and writing Hive tables located in Alluxio on S3, and Alluxio's architecture. Upcoming Alluxio 2.1 features including a Presto connector were also mentioned.

Accelerating Data Computation on Ceph Objects

Alluxio, Inc.

Alluxio Global Online Meetup November 10, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speaker(s): Leonardo Militano, ZHAW In most of the distributed storage systems, the data nodes are decoupled from compute nodes. This is motivated by an improved cost efficiency, storage utilization and a mutually independent scalability of computation and storage. While this consideration is indisputable, several situations exist where moving computation close to the data brings important benefits. Whenever the stored data is to be processed for analytics purposes, all the data needs to be repeatedly moved from the storage to the compute cluster, which leads to reduced performance. In this talk, we will present how using Alluxio computation and storage ecosystems can better interact benefiting the "bringing the data close to the code" approach. Moving away from the complete disaggregation of computation and storage, data locality can enhance the computation performance. During this talk, we will present our observations and testing results that will show important enhancements in accelerating Spark Data Analytics on Ceph Objects Storage using Alluxio.

Data Orchestration for the Hybrid Cloud Era

Alluxio, Inc.

Alluxio Community Office Hour October 20, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speaker(s): Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Alluxio Community Office Hour February 23, 2021 For more Alluxio events: https://www.alluxio.io/events/ Speaker(s): Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Optimizing Latency-sensitive queries for Presto at Facebook: A Collaboration ...

Alluxio, Inc.

Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...

Alluxio, Inc.

ALLUXIO (formerly Tachyon): Unify Data at Memory Speed - Effective using Spar...

Alluxio, Inc.

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio

Alluxio, Inc.

StorageQuery: federated querying on object stores, powered by Alluxio and Presto

Alluxio, Inc.

Alluxio Global Online Meetup August 25, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Abner Ferreira, Simbiose Ventures Caio Pavanelli, Simbiose Ventures Bin Fan, Alluxio Over the last few years, organizations have worked towards the separation of storage and compute for a number of benefits in the areas of cost, data duplication and data latency. Cloud resolves most of these issues but comes to the expense of needing a way to query data on remote storages. Alluxio and Presto are a powerful combination to address the compute problem, which is part of the strategy used by Simbiose Ventures to create a product called StorageQuery - A platform to query files in cloud storages with SQL. This talk will focus on: - How Alluxio fits StorageQuery's tech stack; - Advantages of using Alluxio as a cache layer and its unified filesystem; - Development of new under file system for Backblaze B2 and fine-grained code documentation; - ShannonDB remote storage mode.

Alluxio Use Cases and Future Directions

Alluxio, Inc.

Improving Data Locality for Spark Jobs on Kubernetes Using Alluxio

Alluxio, Inc.

Alluxio Community Office Hour Dec 17, 2019 Speakers: Bin Fan & Jiacheng Liu, Alluxio While adoption of the Cloud & Kubernetes has made it exceptionally easy to scale compute, the increasing spread of data across different systems and clouds has created new challenges for data engineers. Effectively accessing data from AWS S3 or on-premises HDFS becomes harder and data locality is also lost - how do you move data to compute workers efficiently, how do you unify data across multiple or remote clouds, and many more. Open source project Alluxio approaches this problem in a new way. It helps elastic compute workloads, such as Apache Spark, realize the true benefits of the cloud while bringing data locality and data accessibility to workloads orchestrated by Kubernetes. One important performance optimization in Apache Spark is to schedule tasks on nodes with HDFS data nodes locally serving the task input data. However, more users are running Apache Spark natively on Kubernetes where HDFS is not an option. This office hour describes the concept and dataflow with respect to using the stack of Spark/Alluxio in Kubernetes with enhanced data locality even the storage service is outside or remote. In this Office Hour, we will go over: - Why Spark is able to make a locality-aware schedule when working with Alluxio in K8s environment using the host network - Why a pod running Alluxio can share data efficiently with a pod running Spark on the same host using domain socket and host path volume - The roadmap to improve this Spark / Alluxio stack in the context of K8s

Tachyon workshop 2015-07-19

Tachyon Nexus, Inc.

Building Cloud Native Analytical Pipelines on AWS

Alluxio, Inc.

Rise of Intermediate APIs - Beam and Alluxio at Alluxio Meetup 2016

Alluxio, Inc.

This document discusses the rise of intermediary APIs like Apache Beam and Alluxio that allow users to write data processing jobs and express storage lifecycles independently of physical constraints. Intermediary APIs provide portability across frameworks and unified access to multiple storage systems. Alluxio in particular provides an in-memory filesystem that can cache data from various storage sources, while Beam allows processing jobs to run on different execution engines. These intermediary APIs create a path for easy technology adoption and focus on features over connectivity.

Building a high-performance data lake analytics engine at Alibaba Cloud with ...

Alluxio, Inc.

This document discusses optimizations made to Alibaba Cloud's Data Lake Analytics (DLA) engine, which uses Presto, to improve performance when querying data stored in Object Storage Service (OSS). The optimizations included decreasing OSS API request counts, implementing an Alluxio data cache using local disks on Presto workers, and improving disk throughput by utilizing multiple ultra disks. These changes increased cache hit ratios and query performance for workloads involving large scans of data stored in OSS. Future plans include supporting an Alluxio cluster shared by multiple users and additional caching techniques.

Iceberg + Alluxio for Fast Data Analytics

Alluxio, Inc.

Azure Databricks is Easier Than You Think

Ike Ellis

Spark is a fast and general engine for large-scale data processing. It supports Scala, Python, Java, SQL, R and more. Spark applications can access data from many sources and perform tasks like ETL, machine learning, and SQL queries. Azure Databricks provides a managed Spark service on Azure that makes it easier to set up clusters and share notebooks across teams for data analysis. Databricks also integrates with many Azure services for storage and data integration.

Accelerate Spark Workloads on S3

Alluxio, Inc.

This document discusses accelerating Spark workloads on Amazon S3 using Alluxio. It describes the challenges of running Spark interactively on S3 due to its eventual consistency and expensive metadata operations. Alluxio provides a data caching layer that offers strong consistency, faster performance, and API compatibility with HDFS and S3. It also allows data outside of S3 to be analyzed. The document demonstrates how to bootstrap Alluxio on an AWS EMR cluster to accelerate Spark workloads running on S3.

What's hot

Best Practice in Accelerating Data Applications with Spark+Alluxio

Alluxio, Inc.

Ultra Fast Deep Learning in Hybrid Cloud Using Intel Analytics Zoo & Alluxio

Alluxio, Inc.

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Hybrid data lake on google cloud with alluxio and dataproc

Alluxio, Inc.

Accelerating Hive with Alluxio on S3

Alluxio, Inc.

Accelerating Data Computation on Ceph Objects

Alluxio, Inc.

Data Orchestration for the Hybrid Cloud Era

Alluxio, Inc.

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Optimizing Latency-sensitive queries for Presto at Facebook: A Collaboration ...

Alluxio, Inc.

Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...

Alluxio, Inc.

ALLUXIO (formerly Tachyon): Unify Data at Memory Speed - Effective using Spar...

Alluxio, Inc.

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio

Alluxio, Inc.

StorageQuery: federated querying on object stores, powered by Alluxio and Presto

Alluxio, Inc.

Alluxio Use Cases and Future Directions

Alluxio, Inc.

Improving Data Locality for Spark Jobs on Kubernetes Using Alluxio

Alluxio, Inc.

Tachyon workshop 2015-07-19

Tachyon Nexus, Inc.

Building Cloud Native Analytical Pipelines on AWS

Alluxio, Inc.

Rise of Intermediate APIs - Beam and Alluxio at Alluxio Meetup 2016

Alluxio, Inc.

Building a high-performance data lake analytics engine at Alibaba Cloud with ...

Alluxio, Inc.

Iceberg + Alluxio for Fast Data Analytics

Alluxio, Inc.

What's hot (20)

Best Practice in Accelerating Data Applications with Spark+Alluxio

Ultra Fast Deep Learning in Hybrid Cloud Using Intel Analytics Zoo & Alluxio

Accelerate Analytics and ML in the Hybrid Cloud Era

Hybrid data lake on google cloud with alluxio and dataproc

Accelerating Hive with Alluxio on S3

Accelerating Data Computation on Ceph Objects

Data Orchestration for the Hybrid Cloud Era

Accelerate Analytics and ML in the Hybrid Cloud Era

Optimizing Latency-sensitive queries for Presto at Facebook: A Collaboration ...

Enterprise Distributed Query Service powered by Presto & Alluxio across cloud...

ALLUXIO (formerly Tachyon): Unify Data at Memory Speed - Effective using Spar...

Alluxio 2.0 & Near Real-time Big Data Platform w/ Spark & Alluxio

StorageQuery: federated querying on object stores, powered by Alluxio and Presto

Alluxio Use Cases and Future Directions

Improving Data Locality for Spark Jobs on Kubernetes Using Alluxio

Tachyon workshop 2015-07-19

Building Cloud Native Analytical Pipelines on AWS

Rise of Intermediate APIs - Beam and Alluxio at Alluxio Meetup 2016

Building a high-performance data lake analytics engine at Alibaba Cloud with ...

Iceberg + Alluxio for Fast Data Analytics

Similar to Alluxio on AWS EMR Fast Storage Access & Sharing for Spark

Azure Databricks is Easier Than You Think

Ike Ellis

Accelerate Spark Workloads on S3

Alluxio, Inc.

Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics

Alluxio, Inc.

Alluxio Austin Meetup Aug 15, 2019 Speaker: Bin Fan Apache Spark and Alluxio are cousin open source projects that originated from UC Berkeley’s AMPLab. Running Spark with Alluxio is a popular stack particularly for hybrid environments. In this session, I will briefly introduce Apache Spark and Alluxio, share the top ten tips for performance tuning for real-world workloads, and demo Alluxio with Spark.

Building a Cloud Native Stack with EMR Spark, Alluxio, and S3

Alluxio, Inc.

This document summarizes a presentation about building a cloud native stack with EMR Spark, Alluxio, and S3. It discusses using Alluxio to provide better performance than S3 by adding a caching tier and keeping data local to applications like Spark. Alluxio provides familiar file system semantics and can mount multiple data sources. The document demonstrates Alluxio's architecture and how it provides memory speed access to data. It also covers integrating Alluxio with EMR using bootstrap actions and upcoming features in Alluxio 2.0 and 2.1.

Accelerating Analytics with EMR on your S3 Data Lake

Alluxio, Inc.

- Alluxio provides a data caching layer for analytics frameworks like Spark running on AWS EMR, addressing challenges of using S3 directly like inconsistent performance and expensive metadata operations. - It mounts S3 as a unified filesystem and caches frequently used data in memory across workers for faster queries while continuously syncing data to S3. - Alluxio's multi-tier storage enables data to be accessed locally from remote locations like S3 using intelligent policies to promote and demote data between memory, SSDs and disks.

Accelerating Spark with Kubernetes

Alluxio, Inc.

Alluxio can be deployed on Kubernetes to provide data orchestration for analytics frameworks like Spark. Alluxio abstracts data sources and provides a unified namespace, enabling elastic scaling of compute and independent data. It can be deployed with the Alluxio master and workers in separate pods or together with compute frameworks like Spark. A demo was shown of running Spark jobs on Alluxio to get data locality benefits within Kubernetes.

Open Source Data Orchestration for AI, Big Data, and Cloud

Alluxio, Inc.

- Alluxio is an open source data orchestration platform that allows data to be accessed closer to compute across cloud, on-premise, and hybrid environments. - It provides a unified namespace and API to access data located in various storage systems like HDFS, S3, and more. - Alluxio intelligently manages data placement across memory, SSDs, and HDDs for fast data access and supports popular frameworks like Spark, Presto, and Hive.

20181215 introduction to graph databases

Timothy Findlay

This document provides an introduction to graph databases. It defines a graph store as a tool for storing and retrieving highly related data where many things are connected to many other things. It notes that graph databases are optimized for this type of data and discusses some popular graph database implementations. It then explores why graph databases may be useful and some limitations. The document provides examples of graph data modeling and querying capabilities. It also outlines some advanced graph database features and how to interact with a graph database using different programming languages.

Optimizing Data Access for Analytics And AI with Alluxio

Alluxio, Inc.

Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017

Alluxio, Inc.

This document discusses using Alluxio with Spark to improve performance. Alluxio consolidates data in memory across distributed systems to enable faster data sharing between Spark jobs and frameworks. Tests show Alluxio can accelerate Spark workloads by up to 30x when reading from remote storage like S3 by serving data at memory speed. Alluxio also provides data resilience during failures and allows sharing data across jobs more easily.

ML Infrastracture @ Dropbox

Tsahi Glik

The document describes Dropbox's machine learning infrastructure and platform. It discusses how the platform provides scalable access to Dropbox's large data sources for offline and online ML use cases. The platform aims to accelerate ML development at Dropbox by standardizing workflows, automating processes, and making ML deployment and experimentation easy. It utilizes various services like Antenna for activity data and dbxlearn for distributed training across Dropbox and AWS resources. The platform supports all stages of the ML lifecycle from data preparation to model deployment and monitoring.

Building Fast SQL Analytics on Anything with Presto, Alluxio

Alluxio, Inc.

From limited Hadoop compute capacity to increased data scientist efficiency

Alluxio, Inc.

Alluxio Tech Talk Oct 17, 2019 Speaker: Alex Ma, Alluxio Want to leverage your existing investments in Hadoop with your data on-premise and still benefit from the elasticity of the cloud? Like other Hadoop users, you most likely experience very large and busy Hadoop clusters, particularly when it comes to compute capacity. Bursting HDFS data to the cloud can bring challenges – network latency impacts performance, copying data via DistCP means maintaining duplicate data, and you may have to make application changes to accomodate the use of S3. “Zero-copy” hybrid bursting with Alluxio keeps your data on-prem and syncs data to compute in the cloud so you can expand compute capacity, particularly for ephemeral Spark jobs.

Data Orchestration Platform for the Cloud

Alluxio, Inc.

This document discusses using a hybrid cloud approach with data orchestration to enable analytics workloads on data stored both on-premises and in the cloud. It outlines reasons for a hybrid approach including reducing time to production and leveraging cloud flexibility. It then describes alternatives like lift-and-shift or compute-driven approaches and their issues. Finally, it introduces a data orchestration platform that can cache and tier data intelligently while enabling analytics frameworks to access both on-premises and cloud-based data with low latency.

CNCF Member Webinar: Improving Data Locality for Analytics Jobs on Kubernetes...

Alluxio, Inc.

Data processing at the speed of 100 Gbps@Apache Crail (Incubating)

DataWorks Summit

Apache Crail (Incubating) is a distributed data store platform designed to share data at hardware speeds of 100+ Gbps with microsecond access latencies. It uses high-performance user-level I/O, careful software design, and data orchestration techniques to achieve these speeds. Evaluation shows it can deliver full hardware bandwidth and ultra-low access latencies. When used with Apache Spark, it provides significant performance gains for workloads like broadcast, shuffle, TeraSort, and joins. The project is seeking more language and framework integrations and aims to optimize for cloud deployments.

Achieving compute and storage independence for data-driven workloads

Alluxio, Inc.

Alluxio provides a unified interface to access data across multiple storage systems, allowing compute and storage to scale independently for data-driven applications. It uses a virtual unified file system with a global namespace and server-side API translation to abstract data location and access. Alluxio intelligently manages data placement across memory, SSDs and HDDs using multi-tier caching for local performance on remote data. This allows flexible deployment of compute like Spark on any cloud while keeping data fully controlled on-premises. Alluxio is seeing wide adoption with many large production deployments handling thousands of nodes. Upcoming features include POSIX API support and preview of version 2.0.

Enabling Ultra-fast Presto in the Cloud with Alluxio

Alluxio, Inc.

Alluxio is an open source data orchestration system that enables ultra-fast Presto in the cloud. It provides a Presto Alluxio Stack that caches data in Alluxio for faster Presto queries, with benefits like lower latency, more consistent performance, and reduced data transfer. Alluxio's new structured data service provides deeper integration with SQL engines like Presto through features like a catalog service and transformation service. This enables schema-aware optimizations and compute-optimized data formats for further accelerating Presto performance.

Teaching Apache Spark: Demonstrations on the Databricks Cloud Platform

Yao Yao

Yao Yao Mooyoung Lee https://github.com/yaowser/learn-spark/tree/master/Final%20project https://www.youtube.com/watch?v=IVMbSDS4q3A https://www.slideshare.net/YaoYao44/teaching-apache-spark-demonstrations-on-the-databricks-cloud-platform/ Apache Spark is a fast and general engine for big data analytics processing with libraries for SQL, streaming, and advanced analytics Cloud Computing, Structured Streaming, Unified Analytics Integration, End-to-End Applications

Apache Spark 101 - Demi Ben-Ari - Panorays

Demi Ben-Ari

Demi Ben Ari - Apache Spark 101 - First Steps into distributed computing: The world has changed, having one huge server won’t do the job, the ability to Scale Out would be your savior. Apache Spark is a fast and general engine for big data processing, with streaming, SQL, machine learning and graph processing. Showing the basics of Apache Spark and distributed computing. Demi is a Software engineer, Entrepreneur and an International Tech Speaker. Demi has over 10 years of experience in building various systems both from the field of near real time applications and Big Data distributed systems. Co-Founder of the “Big Things” Big Data community and Google Developer Group Cloud. Big Data Expert, but interested in all kinds of technologies, from front-end to backend, whatever moves data around.

Similar to Alluxio on AWS EMR Fast Storage Access & Sharing for Spark (20)

Azure Databricks is Easier Than You Think

Accelerate Spark Workloads on S3

Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics

Building a Cloud Native Stack with EMR Spark, Alluxio, and S3

Accelerating Analytics with EMR on your S3 Data Lake

Accelerating Spark with Kubernetes

Open Source Data Orchestration for AI, Big Data, and Cloud

20181215 introduction to graph databases

Optimizing Data Access for Analytics And AI with Alluxio

Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017

ML Infrastracture @ Dropbox

Building Fast SQL Analytics on Anything with Presto, Alluxio

From limited Hadoop compute capacity to increased data scientist efficiency

Data Orchestration Platform for the Cloud

CNCF Member Webinar: Improving Data Locality for Analytics Jobs on Kubernetes...

Data processing at the speed of 100 Gbps@Apache Crail (Incubating)

Achieving compute and storage independence for data-driven workloads

Enabling Ultra-fast Presto in the Cloud with Alluxio

Teaching Apache Spark: Demonstrations on the Databricks Cloud Platform

Apache Spark 101 - Demi Ben-Ari - Panorays

More from Alluxio, Inc.

AI/ML Infra Meetup | ML explainability in Michelangelo

Alluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Eric Wang (Software Engineer, @Uber) Uber has numerous deep learning models, most of which are highly complex with many layers and a vast number of features. Understanding how these models work is challenging and demands significant resources to experiment with various training algorithms and feature sets. With ML explainability, the ML team aims to bring transparency to these models, helping to clarify their predictions and behavior. This transparency also assists the operations and legal teams in explaining the reasons behind specific prediction outcomes. In this talk, Eric Wang will discuss the methods Uber used for explaining deep learning models and how we integrated these methods into the Uber AI Michelangelo ecosystem to support offline explaining.

AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG

Alluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Junchen Jiang (Assistant Professor of Computer Science, @University of Chicago) Prefill in LLM inference is known to be resource-intensive, especially for long LLM inputs. While better scheduling can mitigate prefill’s impact, it would be fundamentally better to avoid (most of) prefill. This talk introduces our preliminary effort towards drastically minimizing prefill delay for LLM inputs that naturally reuse text chunks, such as in retrieval-augmented generation. While keeping the KV cache of all text chunks in memory is difficult, we show that it is possible to store them on cheaper yet slower storage. By improving the loading process of the reused KV caches, we can still significantly speed up prefill delay while maintaining the same generation quality.

AI/ML Infra Meetup | Perspective on Deep Learning Framework

Alluxio, Inc.

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...

Alluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Data & AI Platform Tech Lead, @Alluxio) - Siyuan Sheng (Senior Software Engineer, @Alluxio) Speed and efficiency are two requirements for the underlying infrastructure for machine learning model development. Data access can bottleneck end-to-end machine learning pipelines as training data volume grows and when large model files are more commonly used for serving. For instance, data loading can constitute nearly 80% of the total model training time, resulting in less than 30% GPU utilization. Also, loading large model files for deployment to production can be slow because of slow network or storage read operations. These challenges are prevalent when using popular frameworks like PyTorch, Ray, or HuggingFace, paired with cloud object storage solutions like S3 or GCS, or downloading models from the HuggingFace model hub. In this presentation, Lu and Siyuan will offer comprehensive insights into improving speed and GPU utilization for model training and serving. You will learn: - The data loading challenges hindering GPU utilization - The reference architecture for running PyTorch and Ray jobs while reading data from S3, with benchmark results of training ResNet50 and BERT - Real-world examples of boosting model performance and GPU utilization through optimized data access

Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud

Alluxio, Inc.

Alluxio Monthly Webinar May. 14, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - ChanChan Mao (Developer Advocate, Alluxio) - Bin Fan (VP of Technology, Alluxio) Running AI/ML workloads in different clouds present unique challenges. The key to a manageable multi-cloud architecture is the ability to seamlessly access data across environments with high performance and low cost. This webinar is designed for data platform engineers, data infra engineers, data engineers, and ML engineers who work with multiple data sources in hybrid or multi-cloud environments. Chanchan and Bin will guide the audience through using Alluxio to greatly simplify data access and make model training and serving more efficient in these environments. You will learn: - How to access data in multi-region, hybrid, and multi-cloud like accessing a local file system - How to run PyTorch to read datasets and write checkpoints to remote storage with Alluxio as the distributed data access layer - Real-world examples and insights from tech giants like Uber, AliPay and more

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Alluxio, Inc.

Alluxio Monthly Webinar Apr. 23, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - ChanChan Mao (Developer Advocate, Alluxio) - Shawn Sun (Tech Lead of Cloud Native, Alluxio) Cloud-native model training jobs require fast data access to achieve shorter training cycles. Accessing data can be challenging when your datasets are distributed across different regions and clouds. Additionally, as GPUs remain scarce and expensive resources, it becomes more common to set up remote training clusters from where data resides. This multi-region/cloud scenario introduces the challenges of losing data locality, resulting in operational overhead, latency and expensive cloud costs. In the third webinar of the multi-cloud webinar series, Chanchan and Shawn dive deep into: - The data locality challenges in the multi-region/cloud ML pipeline - Using a cloud-native distributed caching system to overcome these challenges - The architecture and integration of PyTorch/Ray+Alluxio+S3 using POSIX or RESTful APIs - Live demo with ResNet and BERT benchmark results showing performance gains and cost savings analysis

Speed Up Presto at Uber with Alluxio Caching

Alluxio, Inc.

Correctly Loading Incremental Data at Scale

Alluxio, Inc.

Alluxio x Tobiko - ETL Happy Hour April 16, 2024 For more Alluxio events: https://alluxio.io/events/ Speaker: Toby Mao (CTO @ Tobiko Data) Writing efficient and correct incremental pipelines is challenging. Data practitioners who take on this challenge are viewed as performing an "advanced" function, which discourages broader teams from adopting incremental loads. In this lightning talk, CTO of Tobiko Data, Toby Mao, will demystify incremental loading data at scale.

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio, Inc.

Big Data Bellevue Meetup March 21, 2024 For more Alluxio events: https://alluxio.io/events/ Speakers: Bin Fan (VP of Open Source, Alluxio) In this presentation, Bin Fan (VP of Open Source @ Alluxio) will address a critical challenge of optimizing data loading for distributed Python applications within AI/ML workloads in the cloud, focusing on popular frameworks like Ray and Hugging Face. Integration of Alluxio’s distributed caching for Python applications is accomplished using the fsspec interface, thus greatly improving data access speeds. This is particularly useful in machine learning workflows, where repeated data reloading across slow, unstable or congested networks can severely affect GPU efficiency and escalate operational costs. Attendees can look forward to practical, hands-on demonstrations showcasing the tangible benefits of Alluxio’s caching mechanism across various real-world scenarios. These demos will highlight the enhancements in data efficiency and overall performance of data-intensive Python applications. This presentation is tailored for developers and data scientists eager to optimize their AI/ML workloads. Discover strategies to accelerate your data processing tasks, making them not only faster but also more cost-efficient.

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio, Inc.

Alluxio Monthly Webinar Feb. 27, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer, Alluxio) As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Alluxio, Inc.

Alluxio Monthly Webinar Jan. 30, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Kevin Petrie (VP of Research, Eckerson Group) - Omid Razavi (SVP of Customer Success, Alluxio) 2024 is gearing up to be an impactful year for AI and analytics. Join us on January 30, as Kevin Petrie (VP of Research at Eckerson Group) and Omid Razavi (SVP of Customer Success at Alluxio) share key trends that data and AI leaders should know. This event will efficiently guide you with market data and expert insights to drive successful business outcomes. - Assess current and future trends in data and AI with industry experts - Discover valuable insights and practical recommendations - Learn best practices to make your enterprise data more accessible for both analytics and AI applications

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Juncheng Yang(Ph.D Candidate, @CMU) As a cache eviction algorithm, FIFO has a lot of attractive properties, such as simplicity, speed, scalability, and flash-friendliness. The most prominent criticism of FIFO is its low efficiency (high miss ratio). In this talk, I will describe a simple, scalable FIFO-based algorithm with three static queues (S3-FIFO). Evaluated on 6594 cache traces from 14 datasets, we show that S3- FIFO has lower miss ratios than state-of-the-art algorithms across traces. Moreover, S3-FIFO’s efficiency is robust — it has the lowest mean miss ratio on 10 of the 14 datasets. FIFO queues enable S3-FIFO to achieve good scalability with 6× higher throughput compared to optimized LRU at 16 threads. Our insight is that most objects in skewed workloads will only be accessed once in a short window, so it is critical to evict them early (also called quick demotion). The key of S3-FIFO is a small FIFO queue that filters out most objects from entering the main cache, which provides a guaranteed demotion speed and high demotion precision.

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jingwen Ouyang (Product Manager, @Alluxio) In this session, Jingwen presents an overview of using Alluxio Edge caching to accelerate Trino or Presto queries. She offers practical best practices for using distributed caching with compute engines. In addition, this session also features insights from real-world examples.

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Siyuan Sheng (Senior Software Engineer, @Alluxio) - Chunxu Tang (Research Scientist, @Alluxio) In this session, cloud optimization specialists Chunxu and Siyuan break down the challenges and present a fresh architecture designed to optimize I/O across the data pipeline, ensuring GPUs function at peak performance. The integrated solution of PyTorch/Ray + Alluxio + S3 offers a promising way forward, and the speakers delve deep into its practical applications. Attendees will not only gain theoretical insights but will also be treated to hands-on instructions and demonstrations of deploying this cutting-edge architecture in Kubernetes, specifically tailored for Tensorflow/PyTorch/Ray workloads in the public cloud.

Data Infra Meetup | ByteDance's Native Parquet Reader

Alluxio, Inc.

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jing Zhao (Principal Engineer, @Uber) Uber builds one of the biggest data lakes in the industry, which stores exabytes of data. In this talk, we will introduce the evolution of our data storage architecture, and delve into multiple key initiatives during the past several years. Specifically, we will introduce: - Our on-prem HDFS cluster scalability challenges and how we solved them - Our efficiency optimizations that significantly reduced the storage overhead and unit cost without compromising reliability and performance - The challenges we are facing during the ongoing Cloud migration and our solutions

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

Alluxio, Inc.

Alluxio Monthly Webinar Nov. 15, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer) - Beinan Wang (Senior Staff Engineer & Architect) Many companies are working with development architectures for AI platforms but have concerns about efficiency at scale as data volumes increase. They use centralized cloud data lakes, like S3, to store training data for AI platforms. However, GPU shortages add more complications. Storage and compute can be separate, or even remote, making data loading slow and expensive: 1) Optimizing a developmental setup can include manual copies, which are slow and error-prone 2) Directly transferring data across regions or from cloud to on-premises can incur expensive egress fees This webinar covers solutions to improve data loading for model training. You will learn: - The data loading challenges with distributed infrastructure - Typical solutions, including NFS/NAS on object storage, and why they are not the best options - Common architectures that can improve data loading and cost efficiency - Using Alluxio to accelerate model training and reduce costs

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Adit Madan (Director of Product Management, @Alluxio) In this session, Adit Madan, Director of Product Management at Alluxio, presents an overview of using distributed caching to accelerate model training and serving. He explores the requirements of data access patterns in the ML pipeline and offers practical best practices for using distributed caching in the cloud. This session features insights from real-world examples, such as AliPay, Zhihu, and more.

AI Infra Day | The AI Infra in the Generative AI Era

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (Cheif Architect, VP of Open Source, @Alluxio) As the AI landscape rapidly evolves, the advancements in generative AI technologies, such as ChatGPT, are driving a need for a robust AI infra stack. This opening keynote will explore the key trends of the AI infra stack in the generative AI era.

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Machine Learning Engineer, @Alluxio) - Shawn Sun (Software Engineer, @Alluxio) This hands-on session will discuss best practices for using PyTorch and Alluxio during model training on AWS. Chunxu and Lu will provide a step-by-step demonstration of how to use Alluxio on EKS as a distributed cache to accelerate computer vision model training jobs that read datasets from S3. This architecture significantly improves the utilization of GPUs from 30% to 90%+, archives ~5x faster training, and lower cloud storage costs.

More from Alluxio, Inc. (20)

AI/ML Infra Meetup | ML explainability in Michelangelo

AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG

AI/ML Infra Meetup | Perspective on Deep Learning Framework

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...

Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Speed Up Presto at Uber with Alluxio Caching

Correctly Loading Incremental Data at Scale

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day | The AI Infra in the Generative AI Era

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

Recently uploaded

Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf

VALiNTRY360

Salesforce Healthcare CRM, implemented by VALiNTRY360, revolutionizes patient management by enhancing patient engagement, streamlining administrative processes, and improving care coordination. Its advanced analytics, robust security, and seamless integration with telehealth services ensure that healthcare providers can deliver personalized, efficient, and secure patient care. By automating routine tasks and providing actionable insights, Salesforce Healthcare CRM enables healthcare providers to focus on delivering high-quality care, leading to better patient outcomes and higher satisfaction. VALiNTRY360's expertise ensures a tailored solution that meets the unique needs of any healthcare practice, from small clinics to large hospital systems. For more info visit us https://valintry360.com/solutions/health-life-sciences

fiscal year variant fiscal year variant.

AnkitaPandya11

GreenCode-A-VSCode-Plugin--Dario-Jurisic

Green Software Development

Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)

safelyiotech

Consistent toolbox talks are critical for maintaining workplace safety, as they provide regular opportunities to address specific hazards and reinforce safe practices. These brief, focused sessions ensure that safety is a continual conversation rather than a one-time event, which helps keep safety protocols fresh in employees' minds. Studies have shown that shorter, more frequent training sessions are more effective for retention and behavior change compared to longer, infrequent sessions. Engaging workers regularly, toolbox talks promote a culture of safety, empower employees to voice concerns, and ultimately reduce the likelihood of accidents and injuries on site. The traditional method of conducting safety talks with paper documents and lengthy meetings is not only time-consuming but also less effective. Manual tracking of attendance and compliance is prone to errors and inconsistencies, leading to gaps in safety communication and potential non-compliance with OSHA regulations. Switching to a digital solution like Safelyio offers significant advantages. Safelyio automates the delivery and documentation of safety talks, ensuring consistency and accessibility. The microlearning approach breaks down complex safety protocols into manageable, bite-sized pieces, making it easier for employees to absorb and retain information. This method minimizes disruptions to work schedules, eliminates the hassle of paperwork, and ensures that all safety communications are tracked and recorded accurately. Ultimately, using a digital platform like Safelyio enhances engagement, compliance, and overall safety performance on site. https://safelyio.com/

What next after learning python programming basics

Rakesh Kumar R

原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样

mz5nrf0n

原版一模一样【微信：741003700 】【美国纽约州立大学奥尔巴尼分校毕业证学位证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...

kalichargn70th171

E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies

Quickdice ERP

Measures in SQL (SIGMOD 2024, Santiago, Chile)

Julian Hyde

SQL has attained widespread adoption, but Business Intelligence tools still use their own higher level languages based upon a multidimensional paradigm. Composable calculations are what is missing from SQL, and we propose a new kind of column, called a measure, that attaches a calculation to a table. Like regular tables, tables with measures are composable and closed when used in queries. SQL-with-measures has the power, conciseness and reusability of multidimensional languages but retains SQL semantics. Measure invocations can be expanded in place to simple, clear SQL. To define the evaluation semantics for measures, we introduce context-sensitive expressions (a way to evaluate multidimensional expressions that is consistent with existing SQL semantics), a concept called evaluation context, and several operations for setting and modifying the evaluation context. A talk at SIGMOD, June 9–15, 2024, Santiago, Chile Authors: Julian Hyde (Google) and John Fremlin (Google) https://doi.org/10.1145/3626246.3653374

Modelling Up - DDDEurope 2024 - Amsterdam

Alberto Brandolini

J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...

Bert Jan Schrijver

一比一原版(USF毕业证)旧金山大学毕业证如何办理

dakas1

USF硕士毕业证成绩单【微信95270640】一比一伪造旧金山大学文凭@假冒USF毕业证成绩单+Q微信95270640办理USF学位证书@仿造USF毕业文凭证书@购买旧金山大学毕业证成绩单USF真实使馆认证/真实留信认证回国人员证明 #一整套旧金山大学文凭证件办理#—包含旧金山大学旧金山大学本科毕业证成绩单学历认证|使馆认证|归国人员证明|教育部认证|留信网认证永远存档教育部学历学位认证查询办理国外文凭国外学历学位认证#我们提供全套办理服务。一整套留学文凭证件服务：一：旧金山大学旧金山大学本科毕业证成绩单毕业证 #成绩单等全套材料从防伪到印刷水印底纹到钢印烫金二：真实使馆认证（留学人员回国证明）使馆存档三：真实教育部认证教育部存档教育部留服网站永久可查四：留信认证留学生信息网站永久可查国外毕业证学位证成绩单办理方法： 1客户提供办理旧金山大学旧金山大学本科毕业证成绩单信息：姓名生日专业学位毕业时间等（如信息不确定可以咨询顾问：我们有专业老师帮你查询）； 2开始安排制作毕业证成绩单电子图； 3毕业证成绩单电子版做好以后发送给您确认； 4毕业证成绩单电子版您确认信息无误之后安排制作成品； 5成品做好拍照或者视频给您确认； 6快递给客户（国内顺丰国外DHLUPS等快读邮寄）。教育部文凭学历认证认证的用途：如果您计划在国内发展那么办理国内教育部认证是必不可少的。事业性用人单位如银行国企公务员在您应聘时都会需要您提供这个认证。其他私营 #外企企业无需提供！办理教育部认证所需资料众多且烦琐所有材料您都必须提供原件我们凭借丰富的经验帮您快速整合材料让您少走弯路。实体公司专业为您服务如有需要请联系我: 微信95270640奈一次次令他失望山娃今年岁上五年级识得很多字从走出小屋开始山娃就知道父亲的家和工地共有一个很动听的名字——天河工地的底层空空荡荡很宽阔很凉爽在地上铺上报纸和水泥袋父亲和工人们中午全睡在地上地面坑坑洼洼山娃曾多次绊倒过也曾有长铁钉穿透凉鞋刺在脚板上但山娃不怕工地上也常有五六个从乡下来的小学生他们的父母亲也是高楼上的建筑工人小伙伴来自不同省份都操着带有浓重口音的普通话可不知为啥山娃不仅很快与他们熟识了

Preparing Non - Technical Founders for Engaging a Tech Agency

ISH Technologies

Preparing non-technical founders before engaging a tech agency is crucial for the success of their projects. It starts with clearly defining their vision and goals, conducting thorough market research, and gaining a basic understanding of relevant technologies. Setting realistic expectations and preparing a detailed project brief are essential steps. Founders should select a tech agency with a proven track record and establish clear communication channels. Additionally, addressing legal and contractual considerations and planning for post-launch support are vital to ensure a smooth and successful collaboration. This preparation empowers non-technical founders to effectively communicate their needs and work seamlessly with their chosen tech agency.Visit our site to get more details about this. Contact us today www.ishtechnologies.com.au

Fundamentals of Programming and Language Processors

Rakesh Kumar R

SQL Accounting Software Brochure Malaysia

GohKiangHock

ALGIT - Assembly Line for Green IT - Numbers, Data, Facts

Green Software Development

Energy consumption of Database Management - Florina Jonuzi

Green Software Development

Mobile App Development Company In Noida | Drona Infotech

Drona Infotech

All you need to know about Spring Boot and GraalVM

Alina Yurenko

Unveiling the Advantages of Agile Software Development.pdf

brainerhub1

Recently uploaded (20)

Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf

fiscal year variant fiscal year variant.

GreenCode-A-VSCode-Plugin--Dario-Jurisic

Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)

What next after learning python programming basics

原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样

The Key to Digital Success_ A Comprehensive Guide to Continuous Testing Integ...

E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies

Measures in SQL (SIGMOD 2024, Santiago, Chile)

Modelling Up - DDDEurope 2024 - Amsterdam

J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...

一比一原版(USF毕业证)旧金山大学毕业证如何办理

Preparing Non - Technical Founders for Engaging a Tech Agency

Fundamentals of Programming and Language Processors

SQL Accounting Software Brochure Malaysia

ALGIT - Assembly Line for Green IT - Numbers, Data, Facts

Energy consumption of Database Management - Florina Jonuzi

Mobile App Development Company In Noida | Drona Infotech

All you need to know about Spring Boot and GraalVM

Unveiling the Advantages of Agile Software Development.pdf

Alluxio on AWS EMR Fast Storage Access & Sharing for Spark

1. Alluxio Open Source New York Meetup Alluxio on AWS EMR Fast storage access and sharing for Spark

2. Table of Content ● Spark and Alluxio ● AWS EMR and deployment of Alluxio ● Demo

3. About me Chengzhi Zhao ● Data Platform Engineer at Meetup 2.5 years, 4 years in retail and healthcare ● Area of Interest: Build scalable, reliable and maintainable data infrastructure and tools

4. How do we run spark jobs? EMR Dataproc HDInsight on premise S3 GCS DB HDFS Compute Storage

5. To Summarize... ● Multiple cloud providers ● Storage layer is separated from compute layer ● Data skew/Data Format ● Failure/Restart ● Data sharing is diﬃcult

7. Alluxio

8. Architecture

9. DATA LOCALITY Spark tries to execute tasks as close to the data as possible to minimize data transfer (over the wire). Preferred Locality Order: PROCESS_LOCAL, NODE_LOCAL, RACK_LOCAL, or ANY

10. Check Data Locality

11. Check Data Locality

12. Data Sharing

13. Spark Read & Write spark.read.parquet("alluxio://") val rdd = sc.textFile("alluxio://") df.spark.write.parquet("alluxio://") rdd.saveAsTextFile("alluxio://")

14. Caching ● Avoid duplicated cache ● Crash/Restart

15. Tiered Storage By default, Alluxio only enables a single, memory tier. ● MEM (Memory) ● SSD (Solid State Drives) ● HDD (Hard Disk Drives)

16. Tiered Storage Conﬁgurations alluxio.worker.tieredstore.levels=2 alluxio.worker.tieredstore.level0.alias=MEM alluxio.worker.tieredstore.level0.dirs.quota=24GB alluxio.worker.tieredstore.level0.watermark.high.ratio=0.9 alluxio.worker.tieredstore.level0.watermark.low.ratio=0.7 alluxio.worker.tieredstore.level1.alias=HDD alluxio.worker.tieredstore.level1.dirs.quota=300GB

17. Preload data into Alluxio Uniformly $ bin/alluxio fs load /path/to/load Data Warehouse

18. EMR (Elastic MapReduce) Amazon EMR simplifies building and operating big data environments and applications. EMR features include easy provisioning, scaling, and reconfiguring of clusters, and notebooks for collaborative development. Applications: Flink, Ganglia, Hadoop, HBase, HCatalog, Hive, Hue, JupyterHub, Livy, Mahout, MXNet, Oozie, Phoenix, Pig, Presto, Spark, Sqoop, TensorFlow, Tez, Zeppelin, and ZooKeeper.

19. Setup Alluxio on EMR aws emr create-cluster --applications Name=Spark --release-label emr-5.23.0 --configurations ... --bootstrap-actions '[{"Path":"s3://test/bootstrap_cz.sh"]' --region us-east-1 core-site "fs.alluxio.impl": "alluxio.hadoop.FileSystem" "fs.AbstractFileSystem.alluxio.impl": "alluxio.hadoop.AlluxioFileSystem" spark-defaults "spark.driver.extraClassPath": ":/usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrf s/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/sh are/aws/emr/security/lib/*:/opt/alluxio-1.8.1-hadoop-2.8/client/alluxio-1.8.1-client.jar", "spark.executor.extraClassPath": ":/opt/alluxio-1.8.1-hadoop-2.8/client/alluxio-1.8.1-client.jar"

20. Bootstrap ismaster=`cat /mnt/var/lib/info/instance.json | jq -r '.isMaster'` masterdns=`cat /mnt/var/lib/info/job-flow.json | jq -r '.masterPrivateDnsName'` …. if [[ ${ismaster} == "true" ]]; then sudo ./bin/alluxio bootstrapConf ${masterdns} sudo ./bin/alluxio format sudo ./bin/alluxio-start.sh master else sudo ./bin/alluxio bootstrapConf ${masterdns} initialize_alluxio sudo ./bin/alluxio format sudo ./bin/alluxio-start.sh worker Mount fi ….

21. DEMO

22. Hardware ● r5.2xlarge ● 8 vCore ● 64 GiB memory ● 3 instances

23. Size: 51G Parquet Non Alluxio Alluxio

24. Size: 51G Parquet Non Alluxio Alluxio

25. Q & A https://github.com/ChengzhiZhao/Alluxio-EMR-bootstrap