Alluxio Product School Webinar - Transparent URI

•

0 likes•77 views

Alluxio Product School Webinar Feb. 23, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Greg Palmer (Lead Solutions Engineer, Alluxio) In February’s product school, Greg Palmer, Lead Solution Engineer at Alluxio, will present a live demo featuring Transparent URI, a key feature in Alluxio Enterprise Edition which provides ease of integration of Alluxio with your existing data stack without any changes to the location metadata of the Hive Metastore. Join us to learn the configurations and other advanced settings for employing Transparent URI to simplify DevOps of Alluxio implementation, allowing users to access their existing storage systems without changing URIs at application level.

Software

Alluxio Product School
Webinar - Transparent URI
February 23rd, 2023
Greg Palmer | greg.palmer@alluxio.com | Alluxio
1

Unprecedented Complexity of Data Platforms
3
Data Trend Complex Platform
New compute and storage tech
created every 3-8 years
On-premise, cloud, hybrid,
multi-cloud environments all have
different environment properties
More data generated every day,
and stored in data silos
Data copies, synchronization costs
More people and teams need to
access and leverage these data
Multiple APIs necessitate
integration and application rewrites
3

Data Silos Across Data Centers, Regions and Clouds
Manual copy-based data synchronization across storage systems spread across environments
v
REGION A
v
REGION B
PRIVATE DATA
CENTERS
Amazon
EMR
Cloud
Dataproc
Kubernetes
Engine
Compute
Engine
Hive
DATACENTER 2
DATACENTER 1
ERROR PRONE AND
NETWORK INTENSIVE
DATA COPIES
4
4

Strong Market Demand For Simplification
Acceleration &
auto-tiering of data
based on policies
EFFICIENT ACCESS &
DATA MANAGEMENT
Agility across regions for
private, hybrid or
multi-cloud
ENVIRONMENT
AGNOSTICITY
Multiple APIs to serve
analytics & AI with
storage abstraction
UNIFICATION OF
DATA LAKES
≈
5

Analytics & AI
in the Hybrid & Multi-Cloud Era
Available:
6

No-copy data access across silos
agnostic to compute engine
Foundation of a heterogeneous data
platform across geos
SOLUTION
≈
Multi-Cloud Ready Analytics & AI Platform
v
REGION A
v
REGION B
REGION A REGION B
GKE
DATACENTER 2
DATACENTER 1
HMS
7
7

COMMON LANDING USE CASES
Hybrid Cloud Gateway to utilize
on-prem compute for data in the cloud
CASE 02: HYBRID
Alluxio
Spark
PUBLIC CLOUD
ON PREMISE
Cross Datacenter Access without
changing Ingest Pipeline across regions
CASE 03: MULTI-DATACENTER
Trino/Presto
Alluxio
DATACENTER 1
DATACENTER 2
INGESTION
Consistent SLAs, Performance, and
Cost Savings on cloud storage
CASE 01: CLOUD
PUBLIC CLOUD
Tensorflow
Alluxio
8
Google Cloud Storage

SEAMLESS HIVE CATALOG DEFINITIONS - PRESTO/TRINO
No table redefinitions required using “Transparent URI”
Example Scenario
I. Initial state
A. Data in HDFS
B. Hive Metastore table definitions pointing
to HDFS
II. Compute cluster with Alluxio
A. Catalog points to Hive Metastore
B. Alluxio intercepts Presto/Trino calls to
HDFS
III. Query execution
A. Accesses to HDFS are served by Alluxio
B. No manual data copies or application
re-writes
Presto Catalog
Hive
Metastore
Hive Connector
hdfs://hive/warehouse/table
1.
1I.
Presto/Trino
Alluxio
III.
Public Cloud
On-premise
s
HDFS
10

SEAMLESS HIVE CATALOG DEFINITIONS - SPARK
No table redefinitions required using “Transparent URI”
Example Scenario
I. Initial state
A. Data in HDFS
B. Hive Metastore table definitions pointing
to HDFS
II. Compute cluster with Alluxio
A. Spark points to Hive Metastore
B. Alluxio intercepts Spark calls to HDFS
III. Query execution
A. Accesses to HDFS are served by Alluxio
B. No manual data copies or application
re-writes
Spark Hive Integration
Hive
Metastore
Hive Connector
hdfs://hive/warehouse/table
1.
1I.
Spark
Alluxio
III.
Public Cloud
On-premise
s
HDFS
11

SEAMLESS HIVE CATALOG DEFINITIONS - HDFS & S3
No table redefinitions required using “Transparent URI”
Example Scenario
I. Initial state
A. Data in on-prem HDFS and S3 in cloud
B. Hive Metastore table definitions pointing
to HDFS and S3
II. Compute cluster with Alluxio
A. Spark points to Hive Metastore
B. Alluxio intercepts Spark calls to HDFS
and S3
III. Query execution
A. Accesses to HDFS and S3 are served by
Alluxio
B. No manual data copies or application
re-writes
Hive Integration
Hive
Metastore
Hive Connector
hdfs://hive/warehouse/table
s3:/bucket/hive/warehouse/table
1.
1I.
Trino/Spark
Alluxio
III.
Public Cloud
On-premise
s
HDFS
12
S3

Twitter.com/alluxio
Linkedin.com/alluxio
Website
www.alluxio.io
Slack
http://slackin.alluxio.io/
@
Social Media
Greg Palmer - greg.palmer@alluxio.com
Git Repo: https://github.com/gregpalmr/alluxio-trino-minio

This document discusses modernizing a data platform for analytics and AI across single, hybrid, or multi-cloud environments using Alluxio. It describes Alluxio's key features like data locality, metadata locality, asynchronous data operations, and policy-driven data management that enable consistent performance, portability, and cost savings. Examples are provided of how Alluxio can be used to transition from on-premises HDFS to object storage to hybrid cloud and multi-cloud configurations.

Open Source Data Orchestration for AI, Big Data, and Cloud

Alluxio, Inc.

- Alluxio is an open source data orchestration platform that allows data to be accessed closer to compute across cloud, on-premise, and hybrid environments. - It provides a unified namespace and API to access data located in various storage systems like HDFS, S3, and more. - Alluxio intelligently manages data placement across memory, SSDs, and HDDs for fast data access and supports popular frameworks like Spark, Presto, and Hive.

Modernize Your Data Platform for Analytics and AI

Alluxio, Inc.

Alluxio Product School Webinar Oct. 13, 2022 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: Greg Palmer (Lead Solutions Engineer, Alluxio) Data platform teams are increasingly challenged with accessing multiple data stores that are separated from compute engines, such as Spark, Presto, TensorFlow or PyTorch. Whether your data is distributed across multiple datacenters and/or clouds, a successful heterogeneous data platform requires efficient data access. In October’s Product School, Alluxio’s Lead Solutions Engineer Greg Palmer will present and demo how Alluxio enables you to embrace the cloud migration strategy or multi-cloud architecture for large-scale analytics and AI workloads. Alluxio also helps scale out your platform adoption for analytics and AI across multiple tenants and applications teams.

Introducing the Hub for Data Orchestration

Alluxio, Inc.

Hybrid Data Lake Architecture with Presto & Spark in the cloud accessing on-p...

Alluxio, Inc.

Alluxio Community Office Hour September 29, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speaker(s): Adit Madan, Alluxio In this talk, we describe the architecture to migrate analytics workloads incrementally to any public cloud (AWS, Google Cloud Platform, or Microsoft Azure) directly on on-prem data without copying the data to cloud storage. In this Office Hour: - We will go over an architecture for running elastic compute clusters in the cloud using on-prem HDFS. - Have a casual online video chat with Alluxio Open Source core maintainers to address any Alluxio related questions from our community members

Architecting a Heterogeneous Data Platform Across Clusters, Regions, and Clouds

Alluxio, Inc.

Alluxio foresaw the need for agility when accessing data across silos separated from compute engines like Spark, Presto, Tensorflow and PyTorch. Embracing the separation of storage from compute, the Alluxio data orchestration platform simplifies adoption of the data lake and data mesh paradigm for analytics and AI/ML. In this talk, Bin Fan will share observations to help identify ways to use the platform to meet the needs of your data environment and workloads. 越來越多的企業架構已轉向混合雲和多雲環境。雖然這種轉變帶來了更大的靈活性和敏捷性，但也意味著必須將計算與存儲分離，這就對企業跨框架、跨雲和跨存儲系統的數據管理和編排提出了新的挑戰。此分享將讓聽眾深入了解Alluxio數據編排理念在數據中台對存儲和計算的解耦作用，以及數據編排針對存算分離場景提出的創新架構，同時結合來自金融、運營商、互聯網等行業的典型應用場景來展現Alluxio如何為大數據計算帶來真正的加速，以及如何將數據編排技術用於AI模型訓練！

Meetup at AI NextCon 2019: In-Stream data process, Data Orchestration & More

Alluxio, Inc.

Alluxio 2.0 Deep Dive – Simplifying data access for cloud workloads

Alluxio, Inc.

Alluxio provides a data orchestration platform that allows applications to access data closer to compute across different storage systems through a unified namespace. Key features include intelligent multi-tier caching that provides local performance for remote data, API translation that enables popular frameworks to access different storages without changes, and data elasticity through a global namespace. Alluxio powers analytics and AI workloads in hybrid cloud environments.

Alluxio Day x APAC Modern Data Stack September 22, 2022 For more on Alluxio Day: https://www.alluxio.io/alluxio-day/ For more Alluxio events: https://alluxio.io/events/ Speaker: Bin Fan (Founding Member & VP of Open Source, Alluxio) Alluxio (www.alluxio.io) is an open-source virtual distributed file system that provides a unified data access layer for hybrid and multi-cloud deployments. It enables distributed compute engines like Spark, Presto or Machine Learning frameworks like TensorFlow to transparently access different persistent storage systems (including HDFS, S3, Azure and etc) while actively leveraging in-memory cache to accelerate data access. Developed originally from UC Berkeley AMPLab as research project “Tachyon”, Alluxio has more than 1200 contributors and is used by over 100 companies worldwide with the largest production deployment over 1000 nodes. This presentation focuses on how Alluxio helps the big data analytics stack to be cloud-native. The trending Cloud object storage systems provide more cost-effective and scalable storage solutions but also different semantics and performance implications compared to HDFS. Applications like Spark or Presto will not benefit from the node-level locality or cross-job caching when retrieving data from the cloud object storage. Deploying Alluxio to access cloud solves these problems because data will be retrieved and cached in Alluxio instead of the underlying cloud or object storage repeatedly.

Alluxio Data Orchestration Platform for the Cloud

Shubham Tagra

Alluxio originated as an open source project at UC Berkeley to orchestrate data for cloud applications by providing a unified namespace and intelligent data caching across multiple data sources. It provides consistent high performance for analytics and AI workloads running on object stores by caching frequently accessed data in memory and tiering data to flash/disk based on policies. Alluxio can also enable hybrid cloud environments by allowing on-premises workloads to burst to public clouds without data movement through "zero-copy" access to remote data.

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Alluxio Community Office Hour February 23, 2021 For more Alluxio events: https://www.alluxio.io/events/ Speaker(s): Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Simplified Data Preparation for Machine Learning in Hybrid and Multi Clouds

Alluxio, Inc.

Enabling Apache Spark for Hybrid Cloud

Alluxio, Inc.

1) Alluxio provides a solution for accessing data across hybrid cloud environments by serving as an abstraction layer between applications and underlying storage systems. 2) It allows compute resources and data storage to be separated and scaled independently through its unified namespace and ability to access data locally through intelligent data tiering even when stored remotely. 3) Use cases include bursting big data workloads between on-premise and cloud environments, accelerating popular big data frameworks on object storage, and enabling data orchestration for agility across multiple clouds and storage systems.

Alluxio 2.9 Release Overview

Alluxio, Inc.

Hw09 Hadoop Development At Facebook Hive And Hdfs

Cloudera, Inc.

This document discusses Hadoop and Hive development at Facebook, including how they generate large amounts of user data daily, how they store the data in Hadoop clusters, and how they use Hive as a data warehouse to efficiently run SQL queries on the Hadoop data using a SQL-like language. It also outlines some of Hive's architecture and features like partitioning, buckets, and UDF/UDAF support, as well as its performance improvements over time and future planned work.

Architecting a Heterogeneous Data Platform Across Clusters, Regions, and Clouds

Alluxio, Inc.

Alluxio Product School Webinar January 27, 2022 For more Alluxio events: https://www.alluxio.io/events/ Speaker: Adit Madan Data platform teams are increasingly challenged with accessing multiple data stores that are separated from compute engines, such as Spark, Presto, TensorFlow or PyTorch. Whether your data is distributed across multiple datacenters and/or clouds, a successful heterogeneous data platform requires efficient data access. Alluxio enables you to embrace the separation of storage from compute and use Alluxio data orchestration to simplify adoption of the data lake and data mesh paradigms for analytics and AI/ML workloads. Join Alluxio’s Sr. Product Mgr., Adit Madan, to learn: - Key challenges with architecting a successful heterogeneous data platform - How data orchestration can overcome data access challenges in a distributed, heterogeneous environment - How to identify ways to use Alluxio to meet the needs of your own data environment and workload requirements

Achieving Separation of Compute and Storage in a Cloud World

Alluxio, Inc.

Alluxio Tech Talk Feb 12, 2019 Speaker: Dipti Borkar, Alluxio The rise of compute intensive workloads and the adoption of the cloud has driven organizations to adopt a decoupled architecture for modern workloads – one in which compute scales independently from storage. While this enables scaling elasticity, it introduces new problems – how do you co-locate data with compute, how do you unify data across multiple remote clouds, how do you keep storage and I/O service costs down and many more. Enter Alluxio, a virtual unified file system, which sits between compute and storage that allows you to realize the benefits of a hybrid cloud architecture with the same performance and lower costs. In this webinar, we will discuss: - Why leading enterprises are adopting hybrid cloud architectures with compute and storage disaggregated - The new challenges that this new paradigm introduces - An introduction to Alluxio and the unified data solution it provides for hybrid environments

Future of Data Platform in Cloud Native world

Srivatsan Srinivasan

Presentation of my talk given in Phoenix Data Conference 2019. In this we will look at challenges with current Apache Hadoop ecosystem Apache Hadoop is still relevant but way of doing Hadoop and enterprise data architecture has to be re-looked as we enter Cognitive and Cloud Native Era We need  Architecture that is enabled by common run time layer across on premise and cloud  Architecture that can abstract away dependency and version conflicts with tons of open source machine learning out there. Yarn did not scale up in that aspect until one want to deal with multiple conda environment  Architecture that can enable real Hybrid Cloud and Multi Cloud portability  And many more challenges that one has to overcome to keep architecture simple, infrastructure agile and better utilized

Hadoop and Cloudian HyperStore

Cloudian

The document discusses how Hadoop can work with Cloudian HyperStore, a hybrid cloud storage platform, to provide benefits like in-place analytics without duplicating data, improved efficiency and security, and lower costs. It describes three types of integration - simple access via S3, backup via replication to HyperStore for disaster recovery, and tiering of cold data to HyperStore. Case studies provide examples for each type of integration and how they can solve issues around data and cluster management for customers running Hadoop.

Burst Presto & Spark workloads to AWS EMR with no data copies

Alluxio, Inc.

Alluxio Community Office Hour Apr 28, 2020 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Adit Madan Bin Fan Today’s conventional wisdom states that network latency across the two ends of a hybrid cloud prevents you from running analytic workloads in the cloud with the data on-prem. As a result, most companies copy their data into a cloud environment and maintain that duplicate data. All of this means that it is challenging to make both on-prem HDFS data accessible with the desired application performance. In this talk, we will show you how to leverage any public cloud (AWS, Google Cloud Platform, or Microsoft Azure) to scale analytics workloads directly on on-prem data without copying and synchronizing the data into the cloud. In this Office Hour, we will go over: - A strategy to embrace the hybrid cloud, including an architecture for running ephemeral compute clusters using on-prem HDFS. - An example of running on-demand Presto, Spark, and Hive with Alluxio in the public cloud. - An analysis of experiments with TPC-DS to demonstrate the benefits of the given architecture.

Alluxio @ Uber Seattle Meetup

Alluxio, Inc.

Over the past two decades, the Big Data stack has reshaped and evolved quickly with numerous innovations driven by the rise of many different open source projects and communities. In this meetup, speakers from Uber, Alibaba, and Alluxio will share best practices for addressing the challenges and opportunities in the developing data architectures using new and emerging open source building blocks. Topics include data format (ORC) optimization, storage security (HDFS), data format (Parquet) layers, and unified data access (Alluxio) layers.

Alluxio 2 Community Update

Alluxio, Inc.

Accelerate Spark Workloads on S3

Alluxio, Inc.

This document discusses accelerating Spark workloads on Amazon S3 using Alluxio. It describes the challenges of running Spark interactively on S3 due to its eventual consistency and expensive metadata operations. Alluxio provides a data caching layer that offers strong consistency, faster performance, and API compatibility with HDFS and S3. It also allows data outside of S3 to be analyzed. The document demonstrates how to bootstrap Alluxio on an AWS EMR cluster to accelerate Spark workloads running on S3.

From limited Hadoop compute capacity to increased data scientist efficiency

Alluxio, Inc.

Alluxio Tech Talk Oct 17, 2019 Speaker: Alex Ma, Alluxio Want to leverage your existing investments in Hadoop with your data on-premise and still benefit from the elasticity of the cloud? Like other Hadoop users, you most likely experience very large and busy Hadoop clusters, particularly when it comes to compute capacity. Bursting HDFS data to the cloud can bring challenges – network latency impacts performance, copying data via DistCP means maintaining duplicate data, and you may have to make application changes to accomodate the use of S3. “Zero-copy” hybrid bursting with Alluxio keeps your data on-prem and syncs data to compute in the cloud so you can expand compute capacity, particularly for ephemeral Spark jobs.

Data Orchestration Platform for the Cloud

Alluxio, Inc.

This document discusses using a hybrid cloud approach with data orchestration to enable analytics workloads on data stored both on-premises and in the cloud. It outlines reasons for a hybrid approach including reducing time to production and leveraging cloud flexibility. It then describes alternatives like lift-and-shift or compute-driven approaches and their issues. Finally, it introduces a data orchestration platform that can cache and tier data intelligently while enabling analytics frameworks to access both on-premises and cloud-based data with low latency.

Alluxio Innovations for Structured Data

Alluxio, Inc.

Gene Pang from Alluxio presented on their new structured data management capabilities in Alluxio. Alluxio 2.1.0 includes preview components to integrate SQL engines like Presto with Alluxio's unified metadata catalog and caching. A demo showed Presto queries against a TPCDS dataset on S3 running over 3x faster when using Alluxio's transformations to coalesce and optimize the data format from CSV to Parquet and leverage Alluxio's caching. Future work may include additional connectors, formats, DDL/DML support and client APIs. Feedback from the user community is important to help guide the project.

Alluxio Use Cases and Future Directions

Alluxio, Inc.

How the Development Bank of Singapore solves on-prem compute capacity challen...

Alluxio, Inc.

The Development Bank of Singapore (DBS) has evolved its data platforms over three generations to address big data challenges and the explosion of data. It now uses a hybrid cloud model with Alluxio to provide a unified namespace across on-prem and cloud storage for analytics workloads. Alluxio enables "zero-copy" cloud bursting by caching hot data and orchestrating analytics jobs between on-prem and cloud resources like AWS EMR and Google Dataproc. This provides dynamic scaling of compute capacity while retaining data locality. Alluxio also offers intelligent data tiering and policy-driven data migration to cloud storage over time for cost efficiency and management.

AI/ML Infra Meetup | ML explainability in Michelangelo

Alluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Eric Wang (Software Engineer, @Uber) Uber has numerous deep learning models, most of which are highly complex with many layers and a vast number of features. Understanding how these models work is challenging and demands significant resources to experiment with various training algorithms and feature sets. With ML explainability, the ML team aims to bring transparency to these models, helping to clarify their predictions and behavior. This transparency also assists the operations and legal teams in explaining the reasons behind specific prediction outcomes. In this talk, Eric Wang will discuss the methods Uber used for explaining deep learning models and how we integrated these methods into the Uber AI Michelangelo ecosystem to support offline explaining.

AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG

Alluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Junchen Jiang (Assistant Professor of Computer Science, @University of Chicago) Prefill in LLM inference is known to be resource-intensive, especially for long LLM inputs. While better scheduling can mitigate prefill’s impact, it would be fundamentally better to avoid (most of) prefill. This talk introduces our preliminary effort towards drastically minimizing prefill delay for LLM inputs that naturally reuse text chunks, such as in retrieval-augmented generation. While keeping the KV cache of all text chunks in memory is difficult, we show that it is possible to store them on cheaper yet slower storage. By improving the loading process of the reused KV caches, we can still significantly speed up prefill delay while maintaining the same generation quality.

Similar to Alluxio Product School Webinar - Transparent URI

Unified Data API for Distributed Cloud Analytics and AI

Alluxio, Inc.

Alluxio Data Orchestration Platform for the Cloud

Shubham Tagra

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Simplified Data Preparation for Machine Learning in Hybrid and Multi Clouds

Alluxio, Inc.

Enabling Apache Spark for Hybrid Cloud

Alluxio, Inc.

Alluxio 2.9 Release Overview

Alluxio, Inc.

Hw09 Hadoop Development At Facebook Hive And Hdfs

Cloudera, Inc.

Architecting a Heterogeneous Data Platform Across Clusters, Regions, and Clouds

Alluxio, Inc.

Achieving Separation of Compute and Storage in a Cloud World

Alluxio, Inc.

Future of Data Platform in Cloud Native world

Srivatsan Srinivasan

Hadoop and Cloudian HyperStore

Cloudian

Burst Presto & Spark workloads to AWS EMR with no data copies

Alluxio, Inc.

Alluxio @ Uber Seattle Meetup

Alluxio, Inc.

Alluxio 2 Community Update

Alluxio, Inc.

Accelerate Spark Workloads on S3

Alluxio, Inc.

From limited Hadoop compute capacity to increased data scientist efficiency

Alluxio, Inc.

Data Orchestration Platform for the Cloud

Alluxio, Inc.

Alluxio Innovations for Structured Data

Alluxio, Inc.

Alluxio Use Cases and Future Directions

Alluxio, Inc.

How the Development Bank of Singapore solves on-prem compute capacity challen...

Alluxio, Inc.

Similar to Alluxio Product School Webinar - Transparent URI (20)

Unified Data API for Distributed Cloud Analytics and AI

Alluxio Data Orchestration Platform for the Cloud

Accelerate Analytics and ML in the Hybrid Cloud Era

Simplified Data Preparation for Machine Learning in Hybrid and Multi Clouds

Enabling Apache Spark for Hybrid Cloud

Alluxio 2.9 Release Overview

Hw09 Hadoop Development At Facebook Hive And Hdfs

Architecting a Heterogeneous Data Platform Across Clusters, Regions, and Clouds

Achieving Separation of Compute and Storage in a Cloud World

Future of Data Platform in Cloud Native world

Hadoop and Cloudian HyperStore

Burst Presto & Spark workloads to AWS EMR with no data copies

Alluxio @ Uber Seattle Meetup

Alluxio 2 Community Update

Accelerate Spark Workloads on S3

From limited Hadoop compute capacity to increased data scientist efficiency

Data Orchestration Platform for the Cloud

Alluxio Innovations for Structured Data

Alluxio Use Cases and Future Directions

How the Development Bank of Singapore solves on-prem compute capacity challen...

More from Alluxio, Inc.

AI/ML Infra Meetup | ML explainability in Michelangelo

Alluxio, Inc.

AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG

Alluxio, Inc.

AI/ML Infra Meetup | Perspective on Deep Learning Framework

Alluxio, Inc.

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...

Alluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Data & AI Platform Tech Lead, @Alluxio) - Siyuan Sheng (Senior Software Engineer, @Alluxio) Speed and efficiency are two requirements for the underlying infrastructure for machine learning model development. Data access can bottleneck end-to-end machine learning pipelines as training data volume grows and when large model files are more commonly used for serving. For instance, data loading can constitute nearly 80% of the total model training time, resulting in less than 30% GPU utilization. Also, loading large model files for deployment to production can be slow because of slow network or storage read operations. These challenges are prevalent when using popular frameworks like PyTorch, Ray, or HuggingFace, paired with cloud object storage solutions like S3 or GCS, or downloading models from the HuggingFace model hub. In this presentation, Lu and Siyuan will offer comprehensive insights into improving speed and GPU utilization for model training and serving. You will learn: - The data loading challenges hindering GPU utilization - The reference architecture for running PyTorch and Ray jobs while reading data from S3, with benchmark results of training ResNet50 and BERT - Real-world examples of boosting model performance and GPU utilization through optimized data access

Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud

Alluxio, Inc.

Alluxio Monthly Webinar May. 14, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - ChanChan Mao (Developer Advocate, Alluxio) - Bin Fan (VP of Technology, Alluxio) Running AI/ML workloads in different clouds present unique challenges. The key to a manageable multi-cloud architecture is the ability to seamlessly access data across environments with high performance and low cost. This webinar is designed for data platform engineers, data infra engineers, data engineers, and ML engineers who work with multiple data sources in hybrid or multi-cloud environments. Chanchan and Bin will guide the audience through using Alluxio to greatly simplify data access and make model training and serving more efficient in these environments. You will learn: - How to access data in multi-region, hybrid, and multi-cloud like accessing a local file system - How to run PyTorch to read datasets and write checkpoints to remote storage with Alluxio as the distributed data access layer - Real-world examples and insights from tech giants like Uber, AliPay and more

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Alluxio, Inc.

Alluxio Monthly Webinar Apr. 23, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - ChanChan Mao (Developer Advocate, Alluxio) - Shawn Sun (Tech Lead of Cloud Native, Alluxio) Cloud-native model training jobs require fast data access to achieve shorter training cycles. Accessing data can be challenging when your datasets are distributed across different regions and clouds. Additionally, as GPUs remain scarce and expensive resources, it becomes more common to set up remote training clusters from where data resides. This multi-region/cloud scenario introduces the challenges of losing data locality, resulting in operational overhead, latency and expensive cloud costs. In the third webinar of the multi-cloud webinar series, Chanchan and Shawn dive deep into: - The data locality challenges in the multi-region/cloud ML pipeline - Using a cloud-native distributed caching system to overcome these challenges - The architecture and integration of PyTorch/Ray+Alluxio+S3 using POSIX or RESTful APIs - Live demo with ResNet and BERT benchmark results showing performance gains and cost savings analysis

Optimizing Data Access for Analytics And AI with Alluxio

Alluxio, Inc.

Speed Up Presto at Uber with Alluxio Caching

Alluxio, Inc.

Correctly Loading Incremental Data at Scale

Alluxio, Inc.

Alluxio x Tobiko - ETL Happy Hour April 16, 2024 For more Alluxio events: https://alluxio.io/events/ Speaker: Toby Mao (CTO @ Tobiko Data) Writing efficient and correct incremental pipelines is challenging. Data practitioners who take on this challenge are viewed as performing an "advanced" function, which discourages broader teams from adopting incremental loads. In this lightning talk, CTO of Tobiko Data, Toby Mao, will demystify incremental loading data at scale.

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio, Inc.

Big Data Bellevue Meetup March 21, 2024 For more Alluxio events: https://alluxio.io/events/ Speakers: Bin Fan (VP of Open Source, Alluxio) In this presentation, Bin Fan (VP of Open Source @ Alluxio) will address a critical challenge of optimizing data loading for distributed Python applications within AI/ML workloads in the cloud, focusing on popular frameworks like Ray and Hugging Face. Integration of Alluxio’s distributed caching for Python applications is accomplished using the fsspec interface, thus greatly improving data access speeds. This is particularly useful in machine learning workflows, where repeated data reloading across slow, unstable or congested networks can severely affect GPU efficiency and escalate operational costs. Attendees can look forward to practical, hands-on demonstrations showcasing the tangible benefits of Alluxio’s caching mechanism across various real-world scenarios. These demos will highlight the enhancements in data efficiency and overall performance of data-intensive Python applications. This presentation is tailored for developers and data scientists eager to optimize their AI/ML workloads. Discover strategies to accelerate your data processing tasks, making them not only faster but also more cost-efficient.

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio, Inc.

Alluxio Monthly Webinar Feb. 27, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer, Alluxio) As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Alluxio, Inc.

Alluxio Monthly Webinar Jan. 30, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Kevin Petrie (VP of Research, Eckerson Group) - Omid Razavi (SVP of Customer Success, Alluxio) 2024 is gearing up to be an impactful year for AI and analytics. Join us on January 30, as Kevin Petrie (VP of Research at Eckerson Group) and Omid Razavi (SVP of Customer Success at Alluxio) share key trends that data and AI leaders should know. This event will efficiently guide you with market data and expert insights to drive successful business outcomes. - Assess current and future trends in data and AI with industry experts - Discover valuable insights and practical recommendations - Learn best practices to make your enterprise data more accessible for both analytics and AI applications

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Juncheng Yang(Ph.D Candidate, @CMU) As a cache eviction algorithm, FIFO has a lot of attractive properties, such as simplicity, speed, scalability, and flash-friendliness. The most prominent criticism of FIFO is its low efficiency (high miss ratio). In this talk, I will describe a simple, scalable FIFO-based algorithm with three static queues (S3-FIFO). Evaluated on 6594 cache traces from 14 datasets, we show that S3- FIFO has lower miss ratios than state-of-the-art algorithms across traces. Moreover, S3-FIFO’s efficiency is robust — it has the lowest mean miss ratio on 10 of the 14 datasets. FIFO queues enable S3-FIFO to achieve good scalability with 6× higher throughput compared to optimized LRU at 16 threads. Our insight is that most objects in skewed workloads will only be accessed once in a short window, so it is critical to evict them early (also called quick demotion). The key of S3-FIFO is a small FIFO queue that filters out most objects from entering the main cache, which provides a guaranteed demotion speed and high demotion precision.

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jingwen Ouyang (Product Manager, @Alluxio) In this session, Jingwen presents an overview of using Alluxio Edge caching to accelerate Trino or Presto queries. She offers practical best practices for using distributed caching with compute engines. In addition, this session also features insights from real-world examples.

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Siyuan Sheng (Senior Software Engineer, @Alluxio) - Chunxu Tang (Research Scientist, @Alluxio) In this session, cloud optimization specialists Chunxu and Siyuan break down the challenges and present a fresh architecture designed to optimize I/O across the data pipeline, ensuring GPUs function at peak performance. The integrated solution of PyTorch/Ray + Alluxio + S3 offers a promising way forward, and the speakers delve deep into its practical applications. Attendees will not only gain theoretical insights but will also be treated to hands-on instructions and demonstrations of deploying this cutting-edge architecture in Kubernetes, specifically tailored for Tensorflow/PyTorch/Ray workloads in the public cloud.

Data Infra Meetup | ByteDance's Native Parquet Reader

Alluxio, Inc.

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jing Zhao (Principal Engineer, @Uber) Uber builds one of the biggest data lakes in the industry, which stores exabytes of data. In this talk, we will introduce the evolution of our data storage architecture, and delve into multiple key initiatives during the past several years. Specifically, we will introduce: - Our on-prem HDFS cluster scalability challenges and how we solved them - Our efficiency optimizations that significantly reduced the storage overhead and unit cost without compromising reliability and performance - The challenges we are facing during the ongoing Cloud migration and our solutions

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

Alluxio, Inc.

Alluxio Monthly Webinar Nov. 15, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer) - Beinan Wang (Senior Staff Engineer & Architect) Many companies are working with development architectures for AI platforms but have concerns about efficiency at scale as data volumes increase. They use centralized cloud data lakes, like S3, to store training data for AI platforms. However, GPU shortages add more complications. Storage and compute can be separate, or even remote, making data loading slow and expensive: 1) Optimizing a developmental setup can include manual copies, which are slow and error-prone 2) Directly transferring data across regions or from cloud to on-premises can incur expensive egress fees This webinar covers solutions to improve data loading for model training. You will learn: - The data loading challenges with distributed infrastructure - Typical solutions, including NFS/NAS on object storage, and why they are not the best options - Common architectures that can improve data loading and cost efficiency - Using Alluxio to accelerate model training and reduce costs

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Adit Madan (Director of Product Management, @Alluxio) In this session, Adit Madan, Director of Product Management at Alluxio, presents an overview of using distributed caching to accelerate model training and serving. He explores the requirements of data access patterns in the ML pipeline and offers practical best practices for using distributed caching in the cloud. This session features insights from real-world examples, such as AliPay, Zhihu, and more.

AI Infra Day | The AI Infra in the Generative AI Era

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (Cheif Architect, VP of Open Source, @Alluxio) As the AI landscape rapidly evolves, the advancements in generative AI technologies, such as ChatGPT, are driving a need for a robust AI infra stack. This opening keynote will explore the key trends of the AI infra stack in the generative AI era.

More from Alluxio, Inc. (20)

AI/ML Infra Meetup | ML explainability in Michelangelo

AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG

AI/ML Infra Meetup | Perspective on Deep Learning Framework

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...

Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Optimizing Data Access for Analytics And AI with Alluxio

Speed Up Presto at Uber with Alluxio Caching

Correctly Loading Incremental Data at Scale

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day | The AI Infra in the Generative AI Era

Recently uploaded

Vitthal Shirke Java Microservices Resume.pdf

Vitthal Shirke

LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM

lorraineandreiamcidl

How Can Hiring A Mobile App Development Company Help Your Business Grow?

ToXSL Technologies

Odoo ERP Vs. Traditional ERP Systems – A Comparative Analysis

Envertis Software Solutions

When it is all about ERP solutions, companies typically meet their needs with common ERP solutions like SAP, Oracle, and Microsoft Dynamics. These big players have demonstrated that ERP systems can be either simple or highly comprehensive. This remains true today, but there are new factors to consider, including a promising new contender in the market that’s Odoo. This blog compares Odoo ERP with traditional ERP systems and explains why many companies now see Odoo ERP as the best choice. What are ERP Systems? An ERP, or Enterprise Resource Planning, system provides your company with valuable information to help you make better decisions and boost your ROI. You should choose an ERP system based on your company’s specific needs. For instance, if you run a manufacturing or retail business, you will need an ERP system that efficiently manages inventory. A consulting firm, on the other hand, would benefit from an ERP system that enhances daily operations. Similarly, eCommerce stores would select an ERP system tailored to their needs. Because different businesses have different requirements, ERP system functionalities can vary. Among the various ERP systems available, Odoo ERP is considered one of the best in the ERp market with more than 12 million global users today. Odoo is an open-source ERP system initially designed for small to medium-sized businesses but now suitable for a wide range of companies. Odoo offers a scalable and configurable point-of-sale management solution and allows you to create customised modules for specific industries. Odoo is gaining more popularity because it is built in a way that allows easy customisation, has a user-friendly interface, and is affordable. Here, you will cover the main differences and get to know why Odoo is gaining attention despite the many other ERP systems available in the market.

socradar-q1-2024-aviation-industry-report.pdf

SOCRadar

SOCRadar's Aviation Industry Q1 Incident Report is out now! The aviation industry has always been a prime target for cybercriminals due to its critical infrastructure and high stakes. In the first quarter of 2024, the sector faced an alarming surge in cybersecurity threats, revealing its vulnerabilities and the relentless sophistication of cyber attackers. SOCRadar’s Aviation Industry, Quarterly Incident Report, provides an in-depth analysis of these threats, detected and examined through our extensive monitoring of hacker forums, Telegram channels, and dark web platforms.

Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris

Neo4j

Requirement Traceability in Xen Functional Safety

Ayan Halder

Graspan: A Big Data System for Big Code Analysis

Aftab Hussain

We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs. We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations. These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18. - Accepted in ASPLOS ‘17, Xi’an, China. - Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17. - Invited for presentation at SoCal PLS ‘16. - Invited for poster presentation at PLDI SRC ‘16.

原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样

mz5nrf0n

原版一模一样【微信：741003700 】【美国纽约州立大学奥尔巴尼分校毕业证学位证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

Introducing Crescat - Event Management Software for Venues, Festivals and Eve...

Crescat

Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry. Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events. With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use. Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements. If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io

Energy consumption of Database Management - Florina Jonuzi

Green Software Development

E-commerce Development Services- Hornet Dynamics

Hornet Dynamics

SQL Accounting Software Brochure Malaysia

GohKiangHock

WWDC 2024 Keynote Review: For CocoaCoders Austin

Patrick Weigel

Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf

VALiNTRY360

Salesforce Healthcare CRM, implemented by VALiNTRY360, revolutionizes patient management by enhancing patient engagement, streamlining administrative processes, and improving care coordination. Its advanced analytics, robust security, and seamless integration with telehealth services ensure that healthcare providers can deliver personalized, efficient, and secure patient care. By automating routine tasks and providing actionable insights, Salesforce Healthcare CRM enables healthcare providers to focus on delivering high-quality care, leading to better patient outcomes and higher satisfaction. VALiNTRY360's expertise ensures a tailored solution that meets the unique needs of any healthcare practice, from small clinics to large hospital systems. For more info visit us https://valintry360.com/solutions/health-life-sciences

Mobile App Development Company In Noida | Drona Infotech

Drona Infotech

Atelier - Innover avec l’IA Générative et les graphes de connaissances

Neo4j

Atelier - Innover avec l’IA Générative et les graphes de connaissances Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement. Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.

UI5con 2024 - Bring Your Own Design System

Peter Muessig

How do you combine the OpenUI5/SAPUI5 programming model with a design system that makes its controls available as Web Components? Since OpenUI5/SAPUI5 1.120, the framework supports the integration of any Web Components. This makes it possible, for example, to natively embed own Web Components of your design system which are created with Stencil. The integration embeds the Web Components in a way that they can be used naturally in XMLViews, like with standard UI5 controls, and can be bound with data binding. Learn how you can also make use of the Web Components base class in OpenUI5/SAPUI5 to also integrate your Web Components and get inspired by the solution to generate a custom UI5 library providing the Web Components control wrappers for the native ones.

E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies

Quickdice ERP

Empowering Growth with Best Software Development Company in Noida - Deuglo

Deuglo Infosystem Pvt Ltd

Do you want Software for your Business? Visit Deuglo Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions. Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC). Requirement — Collecting the Requirements is the first Phase in the SSLC process. Feasibility Study — after completing the requirement process they move to the design phase. Design — in this phase, they start designing the software. Coding — when designing is completed, the developers start coding for the software. Testing — in this phase when the coding of the software is done the testing team will start testing. Installation — after completion of testing, the application opens to the live server and launches! Maintenance — after completing the software development, customers start using the software.

Recently uploaded (20)

Vitthal Shirke Java Microservices Resume.pdf

LORRAINE ANDREI_LEQUIGAN_HOW TO USE ZOOM

How Can Hiring A Mobile App Development Company Help Your Business Grow?

Odoo ERP Vs. Traditional ERP Systems – A Comparative Analysis

socradar-q1-2024-aviation-industry-report.pdf

Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris

Requirement Traceability in Xen Functional Safety

Graspan: A Big Data System for Big Code Analysis

原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样

Introducing Crescat - Event Management Software for Venues, Festivals and Eve...

Energy consumption of Database Management - Florina Jonuzi

E-commerce Development Services- Hornet Dynamics

SQL Accounting Software Brochure Malaysia

WWDC 2024 Keynote Review: For CocoaCoders Austin

Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf

Mobile App Development Company In Noida | Drona Infotech

Atelier - Innover avec l’IA Générative et les graphes de connaissances

UI5con 2024 - Bring Your Own Design System

E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies

Empowering Growth with Best Software Development Company in Noida - Deuglo

Alluxio Product School Webinar - Transparent URI

1. Alluxio Product School Webinar - Transparent URI February 23rd, 2023 Greg Palmer | greg.palmer@alluxio.com | Alluxio 1

2. Demands in a Fragmented Data World 2

3. Unprecedented Complexity of Data Platforms 3 Data Trend Complex Platform New compute and storage tech created every 3-8 years On-premise, cloud, hybrid, multi-cloud environments all have different environment properties More data generated every day, and stored in data silos Data copies, synchronization costs More people and teams need to access and leverage these data Multiple APIs necessitate integration and application rewrites 3

4. Data Silos Across Data Centers, Regions and Clouds Manual copy-based data synchronization across storage systems spread across environments v REGION A v REGION B PRIVATE DATA CENTERS Amazon EMR Cloud Dataproc Kubernetes Engine Compute Engine Hive DATACENTER 2 DATACENTER 1 ERROR PRONE AND NETWORK INTENSIVE DATA COPIES 4 4

5. Strong Market Demand For Simplification Acceleration & auto-tiering of data based on policies EFFICIENT ACCESS & DATA MANAGEMENT Agility across regions for private, hybrid or multi-cloud ENVIRONMENT AGNOSTICITY Multiple APIs to serve analytics & AI with storage abstraction UNIFICATION OF DATA LAKES ≈ 5

6. Analytics & AI in the Hybrid & Multi-Cloud Era Available: 6

7. No-copy data access across silos agnostic to compute engine Foundation of a heterogeneous data platform across geos SOLUTION ≈ Multi-Cloud Ready Analytics & AI Platform v REGION A v REGION B REGION A REGION B GKE DATACENTER 2 DATACENTER 1 HMS 7 7

8. COMMON LANDING USE CASES Hybrid Cloud Gateway to utilize on-prem compute for data in the cloud CASE 02: HYBRID Alluxio Spark PUBLIC CLOUD ON PREMISE Cross Datacenter Access without changing Ingest Pipeline across regions CASE 03: MULTI-DATACENTER Trino/Presto Alluxio DATACENTER 1 DATACENTER 2 INGESTION Consistent SLAs, Performance, and Cost Savings on cloud storage CASE 01: CLOUD PUBLIC CLOUD Tensorflow Alluxio 8 Google Cloud Storage

9. ALLUXIO 9 Transparent URI 9

10. SEAMLESS HIVE CATALOG DEFINITIONS - PRESTO/TRINO No table redefinitions required using “Transparent URI” Example Scenario I. Initial state A. Data in HDFS B. Hive Metastore table definitions pointing to HDFS II. Compute cluster with Alluxio A. Catalog points to Hive Metastore B. Alluxio intercepts Presto/Trino calls to HDFS III. Query execution A. Accesses to HDFS are served by Alluxio B. No manual data copies or application re-writes Presto Catalog Hive Metastore Hive Connector hdfs://hive/warehouse/table 1. 1I. Presto/Trino Alluxio III. Public Cloud On-premise s HDFS 10

11. SEAMLESS HIVE CATALOG DEFINITIONS - SPARK No table redefinitions required using “Transparent URI” Example Scenario I. Initial state A. Data in HDFS B. Hive Metastore table definitions pointing to HDFS II. Compute cluster with Alluxio A. Spark points to Hive Metastore B. Alluxio intercepts Spark calls to HDFS III. Query execution A. Accesses to HDFS are served by Alluxio B. No manual data copies or application re-writes Spark Hive Integration Hive Metastore Hive Connector hdfs://hive/warehouse/table 1. 1I. Spark Alluxio III. Public Cloud On-premise s HDFS 11

12. SEAMLESS HIVE CATALOG DEFINITIONS - HDFS & S3 No table redefinitions required using “Transparent URI” Example Scenario I. Initial state A. Data in on-prem HDFS and S3 in cloud B. Hive Metastore table definitions pointing to HDFS and S3 II. Compute cluster with Alluxio A. Spark points to Hive Metastore B. Alluxio intercepts Spark calls to HDFS and S3 III. Query execution A. Accesses to HDFS and S3 are served by Alluxio B. No manual data copies or application re-writes Hive Integration Hive Metastore Hive Connector hdfs://hive/warehouse/table s3:/bucket/hive/warehouse/table 1. 1I. Trino/Spark Alluxio III. Public Cloud On-premise s HDFS 12 S3

13. ALLUXIO 13 Transparent URI Demo 13

14. Twitter.com/alluxio Linkedin.com/alluxio Website www.alluxio.io Slack http://slackin.alluxio.io/ @ Social Media Greg Palmer - greg.palmer@alluxio.com Git Repo: https://github.com/gregpalmr/alluxio-trino-minio

Alluxio Product School Webinar - Transparent URI

Recommended

Recommended

More Related Content

Similar to Alluxio Product School Webinar - Transparent URI

Similar to Alluxio Product School Webinar - Transparent URI (20)

More from Alluxio, Inc.

More from Alluxio, Inc. (20)

Recently uploaded

Recently uploaded (20)

Alluxio Product School Webinar - Transparent URI