Using Alluxio as a Fault-tolerant Pluggable Optimization Component of JD.com's Computation Frameworks

•Download as PPTX, PDF•

2 likes•1,608 views

JD.com is China's largest retailer that uses Alluxio as a fault-tolerant optimization component in its computation frameworks. Alluxio improves JDPresto performance by 10x on 100+ nodes by enabling data caching and reducing remote reads. Ongoing exploration includes running Alluxio on YARN for resource management, using Alluxio as a shuffle service to address disk I/O bottlenecks, and separating computing and storage across clusters for further optimization. JD has also contributed various features and fixes to Alluxio, including a new WebUI, eviction strategies, JVM monitoring, shell commands, and tests.

Using Alluxio as a fault-tolerant pluggable optimization
component of JD.com's computation frameworks
2018-09-13
Bing Bai, JD.com
Tao Huang, JD.com

Introduce JD.com and BDP’s architecture and business
JD and BDP
01
02 The JD use case of Alluxio
JDPresto on Alluxio
03 Alluxio on yarn & shuffle service & storage-computing separation
Ongoing Exploration
Contents

JD BDP Platform
30k+ Node, off-line
cluster 18k+, user
6000+
Cluster
scale
Computing
ability
off-line data daily
40PB+, Job daily
1millon+
450PB+, daily
increase 500TB+
Business
capability
business 40+, data
model 450+
Storage
capacity

JDPresto on Alluxio
JDPresto on Alluxio advantage
Pluggable
Fault-tolerant
Locality
Alluxio can be online or updated at any time, and business’s feeliing is
just a little slow
When we use Alluxio for JDPresto, we make some changes
and bring some good features.
When Alluxio unable to access，JDPresto can access HDFS directly.
Reduce the remote read
• Alluxio led to 10x performance
improvement
• 100+ nodes
• More than 1 year.

JDPresto on Alluxio
Locality
Isolation
load once
use every time
≈ç
AfterBefore

JDPresto on Alluxio
Presto HDFS
Alluxio
Access Alluxio exception
Access HDFS directly
Read HDFS
Data Cache to Alluxio
Read Alluxio

Alluxio on YARN
ResourceM
anager
NodeManager
Alluxio
AppMaster
Alluxio
Worker
Alluxio
Master
Alluxio
Master
NodeManager
Alluxio
Worker
Alluxio
Worker
Client
Spark
Presto
• Unified resource management
• Better elasticity
• Better configuration control and management

Shuffle Service on Alluxio
17
Disk I/O performance bottleneck
Not enough space for the local disk
Executor fails without recalculating
Uniform data TTL ensures that
temporary files are deleted.

Shuffle Service on Alluxio
Shuffle Write phase
Alluxio Node Alluxio Node Alluxio Node
Map Map Map
Shuffle Read phase
Alluxio Cluster
Alluxio Node
ReduceReduce

Shuffle Service on Alluxio
spark-default.conf
spark.shuffle.service.enabled=true
spark.shuffle.store.type = DistributedCache
spark.shuffle.store.distributed.cache.url=alluxio://bdp.jd.com
Implement DistributeCache implemention for shuffle
Re-implement org.apache.spark.shuffle.sort.SortShuffleWriter
Re-implement org.apache.spark.shuffle.sort.HashShuffleReader

Shuffle Service on Alluxio
CPU Usage CPU Usage
TimeTime
Percent
Percent
The comparison between Alluxio FUSE and Alluxio API

Shuffle Service On Alluxio
Using Alluxio FUSE
Using Alluxio API

Separate computing and storage
ResourceM
anager
NodeManager
DataNode
NodeManager
DataNode
NodeManager
DataNode
DataNode DataNode
NodeManagerNodeManager
NameNode
Resource
Manager
Cluster1
Cluster2
Alluxio

JD Contribution to Alluxio
PMC 1
Contributor 6
PR 50
Merged PR 47
Merged Commit 218
Additions/Deletions +4150/-2251

JD Contribution to Alluxio
JD
Contribution
ui-grid based
sort/pagination/filter
add an input field
New WebUI
high watermark start evict
low watermark stop evict
Watermark evict strategy
check startup
check every time
Consistency
monitor JVM pause Periodically
log message and metrics
JVM Pause Monitor
cp/ls/load/rm/
format
Shell Command
DeadLock
thrift add timeout time
…
Bug fix
Shell
RESTful API
Change Log Level
SyncQuery
AlluxioTools
…
Test

Thank You!
baibing3@jd.com
huangtao6@jd.com

Apache Kafka is a new breed of messaging system built for the "big data" world. Coming out of LinkedIn (and donated to Apache), it is a distributed pub/sub system built in Scala. It has been an Apache TLP now for several months with the first Apache release imminent. Built for speed, scalability, and robustness, Kafka should definitely be one of the data tools you consider when designing distributed data-oriented applications. The talk will cover a general overview of the project and technology, with some use cases, and a demo.

Apache Arrow Flight Overview

Jacques Nadeau

Mark Teehan, Principal Solutions Engineer, Confluent Use the Debezium CDC connector to capture database changes from a Postgres database - or MySQL or Oracle; streaming into Kafka topics and onwards to an external data store. Examine how to setup this pipeline using Docker Compose and Confluent Cloud; and how to use various payload formats, such as avro, protobuf and json-schema. https://www.meetup.com/Singapore-Kafka-Meetup/events/276822852/

Cosco: An Efficient Facebook-Scale Shuffle Service

Databricks

Cosco is an efficient shuffle-as-a-service that powers Spark (and Hive) jobs at Facebook warehouse scale. It is implemented as a scalable, reliable and maintainable distributed system. Cosco is based on the idea of partial in-memory aggregation across a shared pool of distributed memory. This provides vastly improved efficiency in disk usage compared to Spark's built-in shuffle. Long term, we believe the Cosco architecture will be key to efficiently supporting jobs at ever larger scale. In this talk we'll take a deep dive into the Cosco architecture and describe how it's deployed at Facebook. We will then describe how it's integrated to run shuffle for Spark, and contrast it with Spark's built-in sort-based shuffle mechanism and SOS (presented at Spark+AI Summit 2018).

Memory Management in Apache Spark

Databricks

Memory management is at the heart of any data-intensive system. Spark, in particular, must arbitrate memory allocation between two main use cases: buffering intermediate data for processing (execution) and caching user data (storage). This talk will take a deep dive through the memory management designs adopted in Spark since its inception and discuss their performance and usability implications for the end user.

分散処理基盤Apache Hadoop入門とHadoopエコシステムの最新技術動向（オープンソースカンファレンス 2015 Tokyo/Spring 講...

NTT DATA OSS Professional Services

Hive spark-s3acommitter-hbase-nfs

Yifeng Jiang

[db tech showcase Tokyo 2016] D32: SPARCサーバ + Pure Storage DB仮想化のすべらない話〜 Exa...

Insight Technology, Inc.

NTTぷらら様は、「柔軟に増減設できるDB基盤」と「コスト最適化」をキーワードに、DB仮想化をSPARCサーバ + Pure Storageの組み合わせで実現しました。更に現在、理想のDB基盤を実現するために、Exadata環境のリプレースも進めています。本セッションでは、検証結果や生のデモンストレーションに、スライドには書けない生々しい話を加え、理想のDB環境実現までの道のりをご紹介します。

Scalable Filesystem Metadata Services with RocksDB

Alluxio, Inc.

Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark

Bo Yang

Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...

Databricks

Parquet is a very popular column based format. Spark can automatically filter useless data using parquet file statistical data by pushdown filters, such as min-max statistics. On the other hand, Spark user can enable Spark parquet vectorized reader to read parquet files by batch. These features improve Spark performance greatly and save both CPU and IO. Parquet is the default data format of data warehouse in Bytedance. In practice, we find that parquet pushdown filters work poorly resulting in reading too much unnecessary data for statistical data has no discrimination across parquet row groups(column data is out of order when writing to parquet files by ETL jobs).

Docker ComposeでMastodonが必要なものを梱包する話

Masahito Zembutsu

HA環境構築のベスト・プラクティス

EnterpriseDB

Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang

Databricks

As a general computing engine, Spark can process data from various data management/storage systems, including HDFS, Hive, Cassandra and Kafka. For flexibility and high throughput, Spark defines the Data Source API, which is an abstraction of the storage layer. The Data Source API has two requirements. 1) Generality: support reading/writing most data management/storage systems. 2) Flexibility: customize and optimize the read and write paths for different systems based on their capabilities. Data Source API V2 is one of the most important features coming with Spark 2.3. This talk will dive into the design and implementation of Data Source API V2, with comparison to the Data Source API V1. We also demonstrate how to implement a file-based data source using the Data Source API V2 for showing its generality and flexibility.

Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...

Databricks

Spark SQL is a highly scalable and efficient relational processing engine with ease-to-use APIs and mid-query fault tolerance. It is a core module of Apache Spark. Spark SQL can process, integrate and analyze the data from diverse data sources (e.g., Hive, Cassandra, Kafka and Oracle) and file formats (e.g., Parquet, ORC, CSV, and JSON). This talk will dive into the technical details of SparkSQL spanning the entire lifecycle of a query execution. The audience will get a deeper understanding of Spark SQL and understand how to tune Spark SQL performance.

Apache Ignite vs Alluxio: Memory Speed Big Data Analytics

DataWorks Summit

Apache Ignite vs Alluxio: Memory Speed Big Data Analytics - Apache Spark’s in memory capabilities catapulted it as the premier processing framework for Hadoop. Apache Ignite and Alluxio, both high-performance, integrated and distributed in-memory platform, takes Apache Spark to the next level by providing an even more powerful, faster and scalable platform to the most demanding data processing and analytic environments. Speaker Irfan Elahi, Consultant, Deloitte

Apache Spark Architecture

Alexey Grishchenko

Deep Dive into Spark SQL with Advanced Performance Tuning

Takuya UESHIN

Using Alluxio as a Fault Tolerant Pluggable Optimization Component to Compute...

Alluxio, Inc.

The Practice of Presto & Alluxio in E-Commerce Big Data Platform

Alluxio, Inc.

What's hot

High-speed Database Throughput Using Apache Arrow Flight SQL

ScyllaDB

40分でわかるHadoop徹底入門（Cloudera World Tokyo 2014 講演資料）

hamaken

A Technical Introduction to WiredTiger

MongoDB

Introduction to Spark Internals

Pietro Michiardi

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

confluent

Cosco: An Efficient Facebook-Scale Shuffle Service

Databricks

Memory Management in Apache Spark

Databricks

分散処理基盤Apache Hadoop入門とHadoopエコシステムの最新技術動向（オープンソースカンファレンス 2015 Tokyo/Spring 講...

NTT DATA OSS Professional Services

Hive spark-s3acommitter-hbase-nfs

Yifeng Jiang

[db tech showcase Tokyo 2016] D32: SPARCサーバ + Pure Storage DB仮想化のすべらない話〜 Exa...

Insight Technology, Inc.

Scalable Filesystem Metadata Services with RocksDB

Alluxio, Inc.

Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark

Bo Yang

Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...

Databricks

Docker ComposeでMastodonが必要なものを梱包する話

Masahito Zembutsu

HA環境構築のベスト・プラクティス

EnterpriseDB

Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang

Databricks

Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...

Databricks

Apache Ignite vs Alluxio: Memory Speed Big Data Analytics

DataWorks Summit

Apache Spark Architecture

Alexey Grishchenko

Deep Dive into Spark SQL with Advanced Performance Tuning

Takuya UESHIN

What's hot (20)

High-speed Database Throughput Using Apache Arrow Flight SQL

40分でわかるHadoop徹底入門（Cloudera World Tokyo 2014 講演資料）

A Technical Introduction to WiredTiger

Introduction to Spark Internals

From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...

Cosco: An Efficient Facebook-Scale Shuffle Service

Memory Management in Apache Spark

分散処理基盤Apache Hadoop入門とHadoopエコシステムの最新技術動向（オープンソースカンファレンス 2015 Tokyo/Spring 講...

Hive spark-s3acommitter-hbase-nfs

[db tech showcase Tokyo 2016] D32: SPARCサーバ + Pure Storage DB仮想化のすべらない話〜 Exa...

Scalable Filesystem Metadata Services with RocksDB

Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark

Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...

Docker ComposeでMastodonが必要なものを梱包する話

HA環境構築のベスト・プラクティス

Apache Spark Data Source V2 with Wenchen Fan and Gengliang Wang

Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...

Apache Ignite vs Alluxio: Memory Speed Big Data Analytics

Apache Spark Architecture

Deep Dive into Spark SQL with Advanced Performance Tuning

Similar to Using Alluxio as a Fault-tolerant Pluggable Optimization Component of JD.com's Computation Frameworks

Using Alluxio as a Fault Tolerant Pluggable Optimization Component to Compute...

Alluxio, Inc.

The Practice of Presto & Alluxio in E-Commerce Big Data Platform

Alluxio, Inc.

HPE Solutions for Challenges in AI and Big Data

Lviv Startup Club

Saviak lviv ai-2019-e-mail (1)

Lviv Startup Club

IBM PureSystems

HelpSystems

This overview of IBM's PureSystems™ family will highlight how key components of IBM Flex System Solutions and PureFlex offerings can save you time and money with: 1. System availability 2. Power consumption 3. Virtualization 4. Multiple platforms and operating systems PureSystems brings together built-in expertise, integrated components, and simplified management to take IT into the next decade. We think that deserves a sigh of relief, and so will you.

Building Fast SQL Analytics on Anything with Presto, Alluxio

Alluxio, Inc.

Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017

Alluxio, Inc.

Fusion IO - Trends and OutlookJonCarvinzer

Storage & Controller Trends and Outlook - August 2013JonCarvinzer

HPC DAY 2017 | HPE Storage and Data Management for Big Data

HPC DAY

How the Development Bank of Singapore solves on-prem compute capacity challen...

Alluxio, Inc.

Alluxio @ Uber Seattle Meetup

Alluxio, Inc.

Over the past two decades, the Big Data stack has reshaped and evolved quickly with numerous innovations driven by the rise of many different open source projects and communities. In this meetup, speakers from Uber, Alibaba, and Alluxio will share best practices for addressing the challenges and opportunities in the developing data architectures using new and emerging open source building blocks. Topics include data format (ORC) optimization, storage security (HDFS), data format (Parquet) layers, and unified data access (Alluxio) layers.

Accelerating Cloud Training With Alluxio

Alluxio, Inc.

Alluxio Day XV September 15, 2022 For more on Alluxio Day: https://www.alluxio.io/alluxio-day/ For more Alluxio events: https://alluxio.io/events/ Speaker: Lu Qiu (Machine Learning Engineer and PMC Maintainer, Alluxio) This talk introduces the three game level progressions to use Alluxio to speed up your cloud training with production use cases from Microsoft, Alibaba, and BossZhipin. - Level 1: Speed up data ingestion from cloud storage - Level 2: Speed up data preprocessing and training workloads - Level 3: Speed up full training workloads with a unified data orchestration layer

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jingwen Ouyang (Product Manager, @Alluxio) In this session, Jingwen presents an overview of using Alluxio Edge caching to accelerate Trino or Presto queries. She offers practical best practices for using distributed caching with compute engines. In addition, this session also features insights from real-world examples.

DDN: Massively-Scalable Platforms and Solutions Engineered for the Big Data a...

inside-BigData.com

Murli Thirumale, CEO Ocarina Networks

EntrepreneurTrek

Key Considerations For Deduplication In The Enterprise

Quantum

robust-company-profile-2015Tecsun Yeep

Leading the Way in Intelligent Data Center Infrastructure

Huawei Enterprise Hong Kong

By Vandia Yang Datacenter Solution Director, Enterprise Business Group, Huawei Hong Kong Traditional data center infrastructure is facing multi-faceted challenges that include high CAPEX, high energy consumption, inefficient operation and management, low availability, and other issues. This session will offer practical solutions, showing how issues are resolved these issues through intelligent data center implementation approach.

Ceph Day Beijing - Storage Modernization with Intel and Ceph

Danielle Womboldt

Similar to Using Alluxio as a Fault-tolerant Pluggable Optimization Component of JD.com's Computation Frameworks (20)

Using Alluxio as a Fault Tolerant Pluggable Optimization Component to Compute...

The Practice of Presto & Alluxio in E-Commerce Big Data Platform

HPE Solutions for Challenges in AI and Big Data

Saviak lviv ai-2019-e-mail (1)

IBM PureSystems

Building Fast SQL Analytics on Anything with Presto, Alluxio

Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017

Fusion IO - Trends and Outlook

Storage & Controller Trends and Outlook - August 2013

HPC DAY 2017 | HPE Storage and Data Management for Big Data

How the Development Bank of Singapore solves on-prem compute capacity challen...

Alluxio @ Uber Seattle Meetup

Accelerating Cloud Training With Alluxio

Data Infra Meetup | Accelerate Your Trino/Presto Queries - Gain the Alluxio Edge

DDN: Massively-Scalable Platforms and Solutions Engineered for the Big Data a...

Murli Thirumale, CEO Ocarina Networks

Key Considerations For Deduplication In The Enterprise

robust-company-profile-2015

Leading the Way in Intelligent Data Center Infrastructure

Ceph Day Beijing - Storage Modernization with Intel and Ceph

More from Alluxio, Inc.

AI/ML Infra Meetup | ML explainability in Michelangelo

Alluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Eric Wang (Software Engineer, @Uber) Uber has numerous deep learning models, most of which are highly complex with many layers and a vast number of features. Understanding how these models work is challenging and demands significant resources to experiment with various training algorithms and feature sets. With ML explainability, the ML team aims to bring transparency to these models, helping to clarify their predictions and behavior. This transparency also assists the operations and legal teams in explaining the reasons behind specific prediction outcomes. In this talk, Eric Wang will discuss the methods Uber used for explaining deep learning models and how we integrated these methods into the Uber AI Michelangelo ecosystem to support offline explaining.

AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG

Alluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Junchen Jiang (Assistant Professor of Computer Science, @University of Chicago) Prefill in LLM inference is known to be resource-intensive, especially for long LLM inputs. While better scheduling can mitigate prefill’s impact, it would be fundamentally better to avoid (most of) prefill. This talk introduces our preliminary effort towards drastically minimizing prefill delay for LLM inputs that naturally reuse text chunks, such as in retrieval-augmented generation. While keeping the KV cache of all text chunks in memory is difficult, we show that it is possible to store them on cheaper yet slower storage. By improving the loading process of the reused KV caches, we can still significantly speed up prefill delay while maintaining the same generation quality.

AI/ML Infra Meetup | Perspective on Deep Learning Framework

Alluxio, Inc.

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...

Alluxio, Inc.

AI/ML Infra Meetup May. 23, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Data & AI Platform Tech Lead, @Alluxio) - Siyuan Sheng (Senior Software Engineer, @Alluxio) Speed and efficiency are two requirements for the underlying infrastructure for machine learning model development. Data access can bottleneck end-to-end machine learning pipelines as training data volume grows and when large model files are more commonly used for serving. For instance, data loading can constitute nearly 80% of the total model training time, resulting in less than 30% GPU utilization. Also, loading large model files for deployment to production can be slow because of slow network or storage read operations. These challenges are prevalent when using popular frameworks like PyTorch, Ray, or HuggingFace, paired with cloud object storage solutions like S3 or GCS, or downloading models from the HuggingFace model hub. In this presentation, Lu and Siyuan will offer comprehensive insights into improving speed and GPU utilization for model training and serving. You will learn: - The data loading challenges hindering GPU utilization - The reference architecture for running PyTorch and Ray jobs while reading data from S3, with benchmark results of training ResNet50 and BERT - Real-world examples of boosting model performance and GPU utilization through optimized data access

Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud

Alluxio, Inc.

Alluxio Monthly Webinar May. 14, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - ChanChan Mao (Developer Advocate, Alluxio) - Bin Fan (VP of Technology, Alluxio) Running AI/ML workloads in different clouds present unique challenges. The key to a manageable multi-cloud architecture is the ability to seamlessly access data across environments with high performance and low cost. This webinar is designed for data platform engineers, data infra engineers, data engineers, and ML engineers who work with multiple data sources in hybrid or multi-cloud environments. Chanchan and Bin will guide the audience through using Alluxio to greatly simplify data access and make model training and serving more efficient in these environments. You will learn: - How to access data in multi-region, hybrid, and multi-cloud like accessing a local file system - How to run PyTorch to read datasets and write checkpoints to remote storage with Alluxio as the distributed data access layer - Real-world examples and insights from tech giants like Uber, AliPay and more

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Alluxio, Inc.

Alluxio Monthly Webinar Apr. 23, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - ChanChan Mao (Developer Advocate, Alluxio) - Shawn Sun (Tech Lead of Cloud Native, Alluxio) Cloud-native model training jobs require fast data access to achieve shorter training cycles. Accessing data can be challenging when your datasets are distributed across different regions and clouds. Additionally, as GPUs remain scarce and expensive resources, it becomes more common to set up remote training clusters from where data resides. This multi-region/cloud scenario introduces the challenges of losing data locality, resulting in operational overhead, latency and expensive cloud costs. In the third webinar of the multi-cloud webinar series, Chanchan and Shawn dive deep into: - The data locality challenges in the multi-region/cloud ML pipeline - Using a cloud-native distributed caching system to overcome these challenges - The architecture and integration of PyTorch/Ray+Alluxio+S3 using POSIX or RESTful APIs - Live demo with ResNet and BERT benchmark results showing performance gains and cost savings analysis

Optimizing Data Access for Analytics And AI with Alluxio

Alluxio, Inc.

Speed Up Presto at Uber with Alluxio Caching

Alluxio, Inc.

Correctly Loading Incremental Data at Scale

Alluxio, Inc.

Alluxio x Tobiko - ETL Happy Hour April 16, 2024 For more Alluxio events: https://alluxio.io/events/ Speaker: Toby Mao (CTO @ Tobiko Data) Writing efficient and correct incremental pipelines is challenging. Data practitioners who take on this challenge are viewed as performing an "advanced" function, which discourages broader teams from adopting incremental loads. In this lightning talk, CTO of Tobiko Data, Toby Mao, will demystify incremental loading data at scale.

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio, Inc.

Big Data Bellevue Meetup March 21, 2024 For more Alluxio events: https://alluxio.io/events/ Speakers: Bin Fan (VP of Open Source, Alluxio) In this presentation, Bin Fan (VP of Open Source @ Alluxio) will address a critical challenge of optimizing data loading for distributed Python applications within AI/ML workloads in the cloud, focusing on popular frameworks like Ray and Hugging Face. Integration of Alluxio’s distributed caching for Python applications is accomplished using the fsspec interface, thus greatly improving data access speeds. This is particularly useful in machine learning workflows, where repeated data reloading across slow, unstable or congested networks can severely affect GPU efficiency and escalate operational costs. Attendees can look forward to practical, hands-on demonstrations showcasing the tangible benefits of Alluxio’s caching mechanism across various real-world scenarios. These demos will highlight the enhancements in data efficiency and overall performance of data-intensive Python applications. This presentation is tailored for developers and data scientists eager to optimize their AI/ML workloads. Discover strategies to accelerate your data processing tasks, making them not only faster but also more cost-efficient.

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio, Inc.

Alluxio Monthly Webinar Feb. 27, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer, Alluxio) As GenAI and AI continue to transform businesses, scaling these workloads requires optimized underlying infrastructure. A multi-cloud architecture allows organizations to leverage different cloud services to meet diverse workload demands while maximizing efficiency, reducing costs, and avoiding vendor lock-in. However, achieving a multi-cloud vision can be challenging. In this webinar, Tarik will share how an agonistic data layer, like Alluxio, allows you to embrace the separation of storage from compute and simplify the adoption of multi-cloud for AI. - Learn why leveraging multiple cloud providers is critical for balancing performance, scalability, and cost of your AI platform - Discover how an agnostic data layer like Alluxio provides seamless data access in multi-cloud that bridges storage and compute without data replication - Gain insights into real-world examples and best practices for deploying AI across on-prem, hybrid, and multi-cloud environments

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Alluxio, Inc.

Alluxio Monthly Webinar Jan. 30, 2024 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Kevin Petrie (VP of Research, Eckerson Group) - Omid Razavi (SVP of Customer Success, Alluxio) 2024 is gearing up to be an impactful year for AI and analytics. Join us on January 30, as Kevin Petrie (VP of Research at Eckerson Group) and Omid Razavi (SVP of Customer Success at Alluxio) share key trends that data and AI leaders should know. This event will efficiently guide you with market data and expert insights to drive successful business outcomes. - Assess current and future trends in data and AI with industry experts - Discover valuable insights and practical recommendations - Learn best practices to make your enterprise data more accessible for both analytics and AI applications

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Juncheng Yang(Ph.D Candidate, @CMU) As a cache eviction algorithm, FIFO has a lot of attractive properties, such as simplicity, speed, scalability, and flash-friendliness. The most prominent criticism of FIFO is its low efficiency (high miss ratio). In this talk, I will describe a simple, scalable FIFO-based algorithm with three static queues (S3-FIFO). Evaluated on 6594 cache traces from 14 datasets, we show that S3- FIFO has lower miss ratios than state-of-the-art algorithms across traces. Moreover, S3-FIFO’s efficiency is robust — it has the lowest mean miss ratio on 10 of the 14 datasets. FIFO queues enable S3-FIFO to achieve good scalability with 6× higher throughput compared to optimized LRU at 16 threads. Our insight is that most objects in skewed workloads will only be accessed once in a short window, so it is critical to evict them early (also called quick demotion). The key of S3-FIFO is a small FIFO queue that filters out most objects from entering the main cache, which provides a guaranteed demotion speed and high demotion precision.

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Siyuan Sheng (Senior Software Engineer, @Alluxio) - Chunxu Tang (Research Scientist, @Alluxio) In this session, cloud optimization specialists Chunxu and Siyuan break down the challenges and present a fresh architecture designed to optimize I/O across the data pipeline, ensuring GPUs function at peak performance. The integrated solution of PyTorch/Ray + Alluxio + S3 offers a promising way forward, and the speakers delve deep into its practical applications. Attendees will not only gain theoretical insights but will also be treated to hands-on instructions and demonstrations of deploying this cutting-edge architecture in Kubernetes, specifically tailored for Tensorflow/PyTorch/Ray workloads in the public cloud.

Data Infra Meetup | ByteDance's Native Parquet Reader

Alluxio, Inc.

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio, Inc.

Data Infra Meetup Jan. 25, 2024 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Jing Zhao (Principal Engineer, @Uber) Uber builds one of the biggest data lakes in the industry, which stores exabytes of data. In this talk, we will introduce the evolution of our data storage architecture, and delve into multiple key initiatives during the past several years. Specifically, we will introduce: - Our on-prem HDFS cluster scalability challenges and how we solved them - Our efficiency optimizations that significantly reduced the storage overhead and unit cost without compromising reliability and performance - The challenges we are facing during the ongoing Cloud migration and our solutions

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

Alluxio, Inc.

Alluxio Monthly Webinar Nov. 15, 2023 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tarik Bennett (Senior Solutions Engineer) - Beinan Wang (Senior Staff Engineer & Architect) Many companies are working with development architectures for AI platforms but have concerns about efficiency at scale as data volumes increase. They use centralized cloud data lakes, like S3, to store training data for AI platforms. However, GPU shortages add more complications. Storage and compute can be separate, or even remote, making data loading slow and expensive: 1) Optimizing a developmental setup can include manual copies, which are slow and error-prone 2) Directly transferring data across regions or from cloud to on-premises can incur expensive egress fees This webinar covers solutions to improve data loading for model training. You will learn: - The data loading challenges with distributed infrastructure - Typical solutions, including NFS/NAS on object storage, and why they are not the best options - Common architectures that can improve data loading and cost efficiency - Using Alluxio to accelerate model training and reduce costs

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Adit Madan (Director of Product Management, @Alluxio) In this session, Adit Madan, Director of Product Management at Alluxio, presents an overview of using distributed caching to accelerate model training and serving. He explores the requirements of data access patterns in the ML pipeline and offers practical best practices for using distributed caching in the cloud. This session features insights from real-world examples, such as AliPay, Zhihu, and more.

AI Infra Day | The AI Infra in the Generative AI Era

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Bin Fan (Cheif Architect, VP of Open Source, @Alluxio) As the AI landscape rapidly evolves, the advancements in generative AI technologies, such as ChatGPT, are driving a need for a robust AI infra stack. This opening keynote will explore the key trends of the AI infra stack in the generative AI era.

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

Alluxio, Inc.

AI Infra Day Oct. 25, 2023 Organized by Alluxio For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Lu Qiu (Machine Learning Engineer, @Alluxio) - Shawn Sun (Software Engineer, @Alluxio) This hands-on session will discuss best practices for using PyTorch and Alluxio during model training on AWS. Chunxu and Lu will provide a step-by-step demonstration of how to use Alluxio on EKS as a distributed cache to accelerate computer vision model training jobs that read datasets from S3. This architecture significantly improves the utilization of GPUs from 30% to 90%+, archives ~5x faster training, and lower cloud storage costs.

More from Alluxio, Inc. (20)

AI/ML Infra Meetup | ML explainability in Michelangelo

AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAG

AI/ML Infra Meetup | Perspective on Deep Learning Framework

AI/ML Infra Meetup | Improve Speed and GPU Utilization for Model Training & S...

Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud

Alluxio Monthly Webinar | Cloud-Native Model Training on Distributed Data

Optimizing Data Access for Analytics And AI with Alluxio

Speed Up Presto at Uber with Alluxio Caching

Correctly Loading Incremental Data at Scale

Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML

Alluxio Monthly Webinar | Why a Multi-Cloud Strategy Matters for Your AI Plat...

Alluxio Monthly Webinar | Five Disruptive Trends that Every Data & AI Leader...

Data Infra Meetup | FIFO Queues are All You Need for Cache Eviction

Data Infra Meetup | Accelerate Distributed PyTorch/Ray Workloads in the Cloud

Data Infra Meetup | ByteDance's Native Parquet Reader

Data Infra Meetup | Uber's Data Storage Evolution

Alluxio Monthly Webinar | Why NFS/NAS on Object Storage May Not Solve Your AI...

AI Infra Day | Accelerate Your Model Training and Serving with Distributed Ca...

AI Infra Day | The AI Infra in the Generative AI Era

AI Infra Day | Hands-on Lab: CV Model Training with PyTorch & Alluxio on Kube...

Recently uploaded

DevOps and Testing slides at DASA Connect

Kari Kakkonen

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

Inflectra

In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring. Learn about: • The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks. • Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective. • Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification. • Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process. Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPathCommunity

💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™: See how to accelerate model training and optimize model performance with active learning Learn about the latest enhancements to out-of-the-box document processing – with little to no training required Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath. Speakers: 👨‍🏫 Andras Palfi, Senior Product Manager, UiPath 👩‍🏫 Lenka Dulovicova, Product Program Manager, UiPath

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Paul Groth

Mission to Decommission: Importance of Decommissioning Products to Increase E...

Product School

IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx

Abida Shariff

UiPath Test Automation using UiPath Test Suite series, part 3

DianaGray10

Designing Great Products: The Power of Design and Leadership by Chief Designe...

Product School

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

Product School

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

Product School

Essentials of Automations: Optimizing FME Workflows with Parameters

Safe Software

Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place. Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects. Here’s what you’ll gain: - Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows. - Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy. - Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency. - Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity. We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic. Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.

The Future of Platform Engineering

Jemma Hussein Allen

JMeter webinar - integration with InfluxDB and Grafana

RTTS

Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application. In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics. Length: 30 minutes Session Overview ------------------------------------------- During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana: - What out-of-the-box solutions are available for real-time monitoring JMeter tests? - What are the benefits of integrating InfluxDB and Grafana into the load testing stack? - Which features are provided by Grafana? - Demonstration of InfluxDB and Grafana using a practice web application To view the webinar recording, go to: https://www.rttsweb.com/jmeter-integration-webinar

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

FIDO Alliance

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Prayukth K V

The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development. The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers: State of global ICS asset and network exposure Sectoral targets and attacks as well as the cost of ransom Global APT activity, AI usage, actor and tactic profiles, and implications Rise in volumes of AI-powered cyberattacks Major cyber events in 2024 Malware and malicious payload trends Cyberattack types and targets Vulnerability exploit attempts on CVEs Attacks on counties – USA Expansion of bot farms – how, where, and why In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East Why are attacks on smart factories rising? Cyber risk predictions Axis of attacks – Europe Systemic attacks in the Middle East Download the full report from here: https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/

Recently uploaded (20)

DevOps and Testing slides at DASA Connect

Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality

GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

To Graph or Not to Graph Knowledge Graph Architectures and LLMs

Mission to Decommission: Importance of Decommissioning Products to Increase E...

IOS-PENTESTING-BEGINNERS-PRACTICAL-GUIDE-.pptx

UiPath Test Automation using UiPath Test Suite series, part 3

Designing Great Products: The Power of Design and Leadership by Chief Designe...

From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

The Art of the Pitch: WordPress Relationships and Sales

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...

Essentials of Automations: Optimizing FME Workflows with Parameters

The Future of Platform Engineering

JMeter webinar - integration with InfluxDB and Grafana

FIDO Alliance Osaka Seminar: Passkeys and the Road Ahead.pdf

State of ICS and IoT Cyber Threat Landscape Report 2024 preview

Using Alluxio as a Fault-tolerant Pluggable Optimization Component of JD.com's Computation Frameworks

1. Using Alluxio as a fault-tolerant pluggable optimization component of JD.com's computation frameworks 2018-09-13 Bing Bai, JD.com Tao Huang, JD.com

2. Introduce JD.com and BDP’s architecture and business JD and BDP 01 02 The JD use case of Alluxio JDPresto on Alluxio 03 Alluxio on yarn & shuffle service & storage-computing separation Ongoing Exploration Contents

3. JD & BDP （Big Data Platform） 2

4. JD Introduction • China’s largest retailer, online or offline • First Chinese internet company to make the Fortune Global 500 list • Strict “zero-tolerance” policy toward counterfeit goods. Customers trust JD because the brand is a guarantee of authenticity 2012 2013 2014 2015 2016 2017 系列 1 Rapid Growth in GMV in Last Six Years* 144.5 billion 93.3 billion 13.4 billion 23.5 billion 46.8 billion Sustained, Rapid Growth 199.1 billion

5. JD BDP Platform 30k+ Node, off-line cluster 18k+, user 6000+ Cluster scale Computing ability off-line data daily 40PB+, Job daily 1millon+ 450PB+, daily increase 500TB+ Business capability business 40+, data model 450+ Storage capacity

6. JD BDP 6

7. JDPresto on Alluxio3

8. JDPresto on Alluxio JDPresto on Alluxio advantage Pluggable Fault-tolerant Locality Alluxio can be online or updated at any time, and business’s feeliing is just a little slow When we use Alluxio for JDPresto, we make some changes and bring some good features. When Alluxio unable to access，JDPresto can access HDFS directly. Reduce the remote read • Alluxio led to 10x performance improvement • 100+ nodes • More than 1 year.

9. JDPresto on Alluxio Locality Isolation load once use every time ≈ç AfterBefore

10. JDPresto on Alluxio Presto HDFS Alluxio Access Alluxio exception Access HDFS directly Read HDFS Data Cache to Alluxio Read Alluxio

11. JDPresto on Alluxio

12. JDPresto on Alluxio 12

13. JDPresto on Alluxio 13 Speed Contrast

14. JDPresto on Alluxio

15. Ongoing Exploration4

16. Alluxio on YARN ResourceM anager NodeManager Alluxio AppMaster Alluxio Worker Alluxio Master Alluxio Master NodeManager Alluxio Worker Alluxio Worker Client Spark Presto • Unified resource management • Better elasticity • Better configuration control and management

17. Shuffle Service on Alluxio 17 Disk I/O performance bottleneck Not enough space for the local disk Executor fails without recalculating Uniform data TTL ensures that temporary files are deleted.

18. Shuffle Service on Alluxio Shuffle Write phase Alluxio Node Alluxio Node Alluxio Node Map Map Map Shuffle Read phase Alluxio Cluster Alluxio Node ReduceReduce

19. Shuffle Service on Alluxio spark-default.conf spark.shuffle.service.enabled=true spark.shuffle.store.type = DistributedCache spark.shuffle.store.distributed.cache.url=alluxio://bdp.jd.com Implement DistributeCache implemention for shuffle Re-implement org.apache.spark.shuffle.sort.SortShuffleWriter Re-implement org.apache.spark.shuffle.sort.HashShuffleReader

20. Shuffle Service on Alluxio CPU Usage CPU Usage TimeTime Percent Percent The comparison between Alluxio FUSE and Alluxio API

21. Shuffle Service On Alluxio Using Alluxio FUSE Using Alluxio API

22. Separate computing and storage ResourceM anager NodeManager DataNode NodeManager DataNode NodeManager DataNode DataNode DataNode NodeManagerNodeManager NameNode Resource Manager Cluster1 Cluster2 Alluxio

23. JD Contribution to Alluxio PMC 1 Contributor 6 PR 50 Merged PR 47 Merged Commit 218 Additions/Deletions +4150/-2251

24. JD Contribution to Alluxio JD Contribution ui-grid based sort/pagination/filter add an input field New WebUI high watermark start evict low watermark stop evict Watermark evict strategy check startup check every time Consistency monitor JVM pause Periodically log message and metrics JVM Pause Monitor cp/ls/load/rm/ format Shell Command DeadLock thrift add timeout time … Bug fix Shell RESTful API Change Log Level SyncQuery AlluxioTools … Test

25. Thank You! baibing3@jd.com huangtao6@jd.com

Using Alluxio as a Fault-tolerant Pluggable Optimization Component of JD.com's Computation Frameworks

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Using Alluxio as a Fault-tolerant Pluggable Optimization Component of JD.com's Computation Frameworks

Similar to Using Alluxio as a Fault-tolerant Pluggable Optimization Component of JD.com's Computation Frameworks (20)

More from Alluxio, Inc.

More from Alluxio, Inc. (20)

Recently uploaded

Recently uploaded (20)

Using Alluxio as a Fault-tolerant Pluggable Optimization Component of JD.com's Computation Frameworks