April Alluxio Community Office Hour Host: Bin Fan Email: community@alluxio.com

1. Office Hour: Alluxio for Hybrid Cloud
2019/04/30 Office Hour
Bin | Founding Engineer | Alluxio

2. 4 big trends driving the need for a new architecture
Separation of
Compute &
Storage
Hybrid – Multi
cloud
environments
Self-service
data across the
enterprise
Rise
of the object
store

3. Data Ecosystem - Beta Data Ecosystem 1.0
COMPUTE
STORAGE STORAGE
COMPUTE

4. Co-located
Big data journey and innovation options for enterprises
Co-located
compute & HDFS
on the same cluster
Disaggregated
compute & HDFS
on the same cluster
MR / Hive
HDFS
Hive
HDFS
Disaggregated
Burst HDFS data in
the cloud,
public or private
Support Presto, Spark
and other computes
without app changes
Enable & accelerate
big data on
object stores
Transition to Object store
HDFS for Hybrid Cloud
Support more frameworks

5. ▪ Accessing data over WAN too
slow
▪ Copying data to compute cloud
time consuming and complex
▪ Using another storage system like
S3 means expensive application
changes
▪ Using S3 via HDFS connector
leads to extremely low
performance
Challenges with the transition
HDFS for Hybrid Cloud
▪ Copying data to multiple
compute clouds time consuming
and error prone
▪ Migrating applications for new
storage systems is complex &
time consuming
▪ Storing and managing multiple
copies of the data becomes
expensive
Support more frameworks
▪ Object stores performance for
big data workloads can be very
poor
▪ No native support for popular
frameworks
▪ Expensive metadata operations
reduce performance even more
▪ No support for hybrid
environments directly
Transition to Object store

6. Data Orchestration for the Cloud
Java File API HDFS Interface S3 Interface REST APIPOSIX Interface
HDFS Driver Swift Driver S3 Driver NFS Driver
Independent scaling of compute & data

7. Use Cases Data Orchestration Enables
Hive
Alluxio
Burst big data workloads in
hybrid cloud environments
On premise
Same instance /
container
Alluxio
On-premise
PrestoSpark
Alluxio
Accelerate big data frameworks
on the public cloud
Same instance /
container
Dramatically speed-up big data
on object stores on premise
Same container
/ machine
or or

8. Advanced Use Cases
Spark
Alluxio
Any Cloud / Multi Cloud
Same data
center / region
Presto
Enable big data on object stores
across single or multiple clouds
Standalone
Spark
Alluxio
Orchestrate data frameworks on
the public cloud
Any public /
private cloud
or or
PrestoHive

9. Data Elasticity
with a unified
namespace
Abstract data silos & storage
systems to independently scale
data on-demand with compute
Run Spark, Hive, Presto, ML
workloads on your data
located anywhere
Accelerate big data
workloads with transparent
tiered local data
Data Accessibility
for popular APIs &
API translation
Data Locality
with Intelligent
Multi-tiering
Alluxio – Key innovations

10. Data Locality with Intelligent Multi-tiering
Local performance from remote data using multi-tier storage
Hot Warm Cold
RAM SSD HDD
Read & Write Buffering
Transparent to App
Policies for pinning,
promotion/demotion, TTL

11. Data Accessibility via popular APIs and API Translation
Convert from Client-side Interface to native Storage Interface
Java File API HDFS Interface S3 Interface REST APIFUSE Interface
HDFS Driver Swift DriverS3 Driver NFS Driver

12. Data Elasticity via Unified Namespace
Enables effective data management across different Under Store
- Uses Mounting with Transparent Naming

13. Unified Namespace: Global Data Accessibility
Transparent access to understorage makes all enterprise data
available locally
SUPPORTS
• HDFS
• NFS
• OpenStack
• Ceph
• Amazon S3
• Azure
• Google Cloud
IT OPS FRIENDLY
• Storage mounted into Alluxio
by central IT
• Security in Alluxio mirrors
source data
• Authentication through
LDAP/AD
• Wireline encryption
HDFS #1
Object Store
NFS
HDFS #2

14. DATA ORCHESTRATION
SPARK
HDFS
SPARK
HDFS
Public Cloud
Public Cloud
▪ Compute scales elastically independent of storage
▪ Faster time to insights with seamless data
orchestration
▪ Accelerated workloads with memory-first data
approach
Leading Hedge Fund
Fastest growing big hedge fund managing $46 billion for investors
Use case | Cloud bursting on-premise data

15. Use case | Data orchestration for agility
DATA ORCHESTRATION
SPARK
HDFS
SPARK
Kubernetes
OBJECT HBASE
ETLSPARK
HDFS OBJECT HBASE
▪ Single namespace to access & address all data
▪ Data local to compute accelerates workloads
China Unicom
Leading Chinese Telco serving 320 million subscribers

16. Customer Insights Use Case – Top
Telecom
Challenge –
Desired a central view of consumer
information in near real time for
proactive support.
Many HDFS, different distributions,
many incompatible versions.
On-prem & cloud. Integration
through heavy ETL.
Solution –
Alluxio integrates data into central
catalog for fast access to consumer
interaction records.
Impact –
Reduced integration time
Faster data speed & freshness
HADOOP ML HADOOP
HDFS HDFS HDFS
ML
ETL
HDP
HDFS
CDH
HDFS
MAPR
HDFS
HDFS

17. Enterprises moving towards independent compute & storage

18. Incredible Open Source Momentum with growing community
900+ contributors &
growing
3760+ Git Stars
Apache 2.0 Licensed
Hundreds of thousands
of downloads
Join the conversation on Slack
alluxio.org/slack

19. Questions?
Join the Alluxio Community
www.alluxio.org | www.alluxio.com | Twitter: @alluxio