1.
The Next Step
Yosuke Hara - @yosukehara
A Researcher of R.I.T. and Tech Lead LeoFS
with Paras Patel, An engineer of Rakuten DU
1

2.
Overview
Brief Benchmark Report
v1.0 -Multi Data Center Replication
v1.1 - NFS Support
LeoFS Administration at Rakuten
Future Plan
2

3.
Overview
3

4.
2008 2010 2012 2016 2018
2014 2020
LeoFS - Background
Terabyte Petabyte Exabyte
4

5.
The Lion of Storage Systems
HIGH Availability
3 Vs in 3 HIGHs
HIGH Cost
Performance Ratio
HIGH
Scalability
LeoFS Non Stop
Velocity: Low Latency
Minimum Resources
Volume: Petabyte / Exabyte
Variety: Photo, Movie, Unstructured-data
5

6.
Storage Engine/Router
Metadata Object Storage
Monitor
GUI Console
( Erlang RPC)
LeoFS - Architecture
Storage
(storage cluster)
Manager
( Erlang RPC)
Gateway
( TCP/IP,SNMP )
S3-API / NFS
- Software Defined Storage
- Satisfied the 3 HIGH
- Easy Administration
Storage Engine/Router
Metadata Object Storage
Storage Engine/Router
Metadata Object Storage
Clients
.....
Monitoring
System
Huge Capacity Storage
6

7.
LeoFS Gateway
7

8.
LeoFS Overview - Gateway
HTTP Request and Response
Built in Object Cache Mechanism
REST-API / S3-API
Stateless Proxy + Object Cache
[ Memory Cache, Disc Cache ]
Use Consistent Hashing
for decision of a primary node
Clients Gateway(s) Storage Cluster
Storage Cluster
Fast HTTP Server - Cowboy
API Handler
Object Cache Mechanism
8

9.
LeoFS Storage
9

10.
LeoFS Overview - Storage
Gateway Storage (Storage Cluster)
WRITE: Auto Replication
READ : Auto Repair of an Inconsistent Object with Async
Use Consistent Hashing
for Data Operation
in the Storage Cluster
Choosing Replica Target Node(s)
RING
2 ^ 128 (MD5)
# of replicas = 3
KEY = “bucket/leofs.key”
Hash = md5(Filename)
Secondary-1
Secondary-2
Primary Node
P2P
10

11.
Storage consists of Object Storage and Metadata Storage
Includes Replicator and Recoverer for the eventual consistency
Request From Gateway
LeoFS Overview - Storage
...
LeoFS Storage
Replicator
Recoverer
...
Storage Engine
Storage Engine, Metadata Gateway + Object Storage
Metadata
Storage Object
Storage
11

12.
LeoFS Overview - Storage - Data Structure
Metadata
Storage
Object Storage
for retrieving an object
Offset Version Time-
Key stamp
for Sync
Checksum KeySize DataSize Offset Version Time-stamp
Key User-Meta Footer
Actual
File
Robust and
High Performance
Necessary for GC
Metadata
Checksum
KeySize Custom
Meta Size File Size
Footer (8B)
User-Meta
Size
Header (Metadata - Fixed length) Body (Variable Length)
Needle
Super-block
Needle-1
Needle-2
Needle-3
Object Container
Needle-4
Needle-5
Log Structure File Format
12

13.
LeoFS Manager
13

14.
LeoFS Overview - Manager
Storage Cluster
Monitor
RING, Node State
Operate
status, suspend,
resume, detach,
whereis, ...
Gateway(s)
Gateway(s) Storage Cluster
Manager(s)
Operates LeoFS - Gateway and Storage Cluster
RING Monitor and NodeState Monitor
14

15.
Brief Benchmark
Report
15

16.
Brief Benchmark Report
1st Case:
Group of Value Ranges
Storage:5, Gateway:1, Manager:2
R:W = 9:1
source: https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/20140605/tests/1m_r9w1_240min
2nd Case:
Group of Value Ranges
Storage:5, Gateway:1, Manager:2
R:W = 8:2
source: https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/20140605/tests/1m_r8w2_120min
16

17.
Brief Benchmark Report
Server Spec - Gateway:
CPU Intel(R) Xeon(R) CPU X5650 @ 2.67GHz * 2 (12 cores / 24 threads)
Memory 96GB
Disk HDD - 240GB RAID0
Network 10G-Ether
Server Spec - Storage x5:
CPU Intel(R) Xeon(R) CPU X5650 @ 2.67GHz * 2 (12 cores / 24 threads)
Memory 96GB
Disk
HDD - 240GB RAID0 (System)
HDD - 2TB RAID0 (Data)
Network 10G-Ether
17

18.
Brief Benchmark Report - 1st Case (R:W=9:1)
Environment:
Network 10Gbps
OS CentOS release 6.5 (Final)
Erlang OTP R16B03-1
LeoFS v1.0.2
System Consistency Level: [ N:3, W:2, R:1, D:2 ]
Benchmark Configuration:
Duration 4.0h
R:W 9:1
# of Concurrent Processes 64
# of Keys 100,000
Value Size
Range (byte) Percentage
1,024 10,240 24%
10,241 102,400 30%
102,401 819,200 30%
819,201 1,572,864 16%
18

19.
Brief Benchmark Report - 1st Case (R:W=9:1)
The range of the data size of 80%
is between 1KB and 1.5MB
Range (byte) Percentage
1,024 10,240 24%
10,241 102,400 30%
102,401 819,200 30%
819,201 1,572,864 16%
19

20.
Brief Benchmark Report - 1st Case (R:W=9:1)
source: https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/20140601/tests/1m_r9w1_240min
50ms
50ms
1,500ops
No Errors
OPS Latency
20

21.
Brief Benchmark Report - 1st Case / Network Traffic
1,500,000
1,350,000
1,200,000
1,050,000
900,000
750,000
600,000
450,000
300,000
150,000
0
0s
500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
5000s
5500s
6000s
6500s
6.0Gbps
7000s
7500s
8000s
8500s
9000s
9500s
10000s
10500s
gateway rxbyt/s gateway txbyt/s
storage-1 rxbyt/s storage-1 txbyt/s
storage-2 rxbyt/s storage-2 txbyt/s
storage-3 rxbyt/s storage-3 txbyt/s
storage-4 rxbyt/s storage-4 txbyt/s
storage-5 rxbyt/s storage-5 txbyt/s
11000s
11500s
12000s
12500s
13000s
13500s
14000s
10.0Gbps
7.0Gbps
5.0Gbps
Storage Gateway
60%
21

22.
Brief Benchmark Report - 1st Case / Memory and CPU
14000s
13500s
13000s
12500s
12000s
11500s
11000s
10500s
10000s
9500s
9000s
8500s
8000s
7500s
7000s
6500s
6000s
5500s
5000s
4500s
4000s
3500s
3000s
2500s
2000s
1500s
1000s
500s
0
0.1
0.3
0.4
0.6
0.7
0.9
1.0
1.1
1.3
1.4
1.6
1.7
1.9
2.0
0s
Memory Usage
CPU Load 5min
1.0
14000s
13500s
13000s
12500s
12000s
11500s
11000s
10500s
10000s
9500s
9000s
8500s
8000s
7500s
7000s
6500s
6000s
5500s
5000s
4500s
4000s
3500s
3000s
2500s
2000s
1500s
1000s
500s
0
10
20
30
40
50
60
70
80
90
100
0s
gateway storage-1
storage-2 storage-3
storage-4 storage-5
22

23.
Brief Benchmark Report - 2nd Case (R:W=8:2)
Environment:
Network 10Gbps
OS CentOS release 6.5 (Final)
Erlang OTP R16B03-1
LeoFS v1.0.2
System Consistency Level: [ N:3, W:2, R:1, D:2 ]
Benchmark Configuration:
Duration 2.0h
R:W 8:2
# of Concurrent Processes 64
# of Keys 100,000
Value Size
Range (byte) Percentage
1,024 10,240 24%
10,241 102,400 30%
102,401 819,200 30%
819,201 1,572,864 16%
23

24.
Brief Benchmark Report - 2nd Case (R:W=8:2)
65ms 85ms
1,000ops
No Errors
OPS Latency
24

25.
Compare 1st case
with 2nd case
25

26.
Brief Benchmark Report
1,500,000
1,350,000
1,200,000
1,050,000
900,000
750,000
600,000
450,000
300,000
150,000
0
0s
500s
1000s
1500s
2000s
2500s
3000s
7.0Gbps
3500s
4000s
4500s
5000s
gateway rxbyt/s gateway txbyt/s
storage-1 rxbyt/s storage-1 txbyt/s
storage-2 rxbyt/s storage-2 txbyt/s
storage-3 rxbyt/s storage-3 txbyt/s
storage-4 rxbyt/s storage-4 txbyt/s
storage-5 rxbyt/s storage-5 txbyt/s
5500s
6000s
6500s
7000s
3,000,000
2,700,000
2,400,000
2,100,000
1,800,000
1,500,000
1,200,000
900,000
600,000
300,000
0
0s
500s
1000s
1500s
2000s
2500s
3000s
minus 0.7Gbps
7.0Gbps
3500s
4000s
4500s
5000s
5500s
6000s
6500s
7000s
6.0Gbps
6.0Gbps
1st Case - Network Traffic
2nd Case - Network Traffic
26

27.
Brief Benchmark Report
600.0
550.0
500.0
450.0
400.0
350.0
300.0
250.0
200.0
150.0
100.0
50.0
600.0
550.0
500.0
450.0
400.0
350.0
300.0
250.0
200.0
150.0
100.0
50.0
0
0s
500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
storage-1 storage-2 storage-3
storage-4 storage-5
5000s
5500s
200
6000s
6500s
7000s
0
0s
500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
5000s
5500s
6000s
6500s
7000s
100
100
2nd Case - Disk util%
200
1st Case - Disk util%
1.8x high
27

28.
Brief Benchmark Report
1st Case - CPU Load 5min
3.0
2.8
2.6
2.4
2.2
2.0
1.8
1.6
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0
0s
500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
5000s
1.00
5500s
6000s
6500s
7000s
gateway
storage-1
storage-2
storage-3
storage-4
storage-5
3.0
2.8
2.6
2.4
2.2
2.0
1.8
1.6
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0
0s
500s
1000s
1500s
2000s
2500s
3000s
3500s
4000s
4500s
5000s
5500s
6000s
6500s
7000s
1.00
1.6x high
2nd Case - CPU Load 5min
28

29.
Brief Benchmark Report
Conclusion:
LeoFS kept in a stable performance
through the benchmark
Bottleneck is Disk I/O
The cache mechanism contributed to reduce
network traffic between Gateway and Storage
29

30.
LeoFS v1.0
Multi Data Center
Replication
30

31.
LeoFS - For Disaster Recovery
Storage cluster Manager cluster
For Disaster Recovery
= High Scalability + High Availability
w/o SPOF and Performance Degradation
31

32.
Multi Data Center Replication
Designes it as simple as possible
1. Easy Operation to build multi clusters.
2. Asynchronous data replication between clusters
Stacked data is transferred to remote cluster(s)
3. Eventual consistency
32

33.
join cluster DC-2 and DC-3
[# of replicas:3] [# of replicas:1] [# of replicas:1]
DC-2 DC-3
Client Storage cluster Manager cluster
DC-1
leo_rpc leo_rpc
Monitors and Replicates each “RING” and “System Configuration”
Leo Storage Platform
Multi Data Center Replication
Executing “Join Cluster”
on Manager Console
Preparing the MDC Replication
33

34.
[# of replicas:1] [# of replicas:1]
DC-2 DC-3
Multi Data Center Replication
Stacking objects
Client Storage cluster Manager cluster
Monitors and Replicates each “RING” and “System Configuration”
Leo Storage Platform
Application(s)
Request to
the Target Region
[# of replicas:3]
DC-1
Temporally Stacking objects
- One container's capacity is *32MB
- When capacity is full,
send it to remote cluster(s)
* 32MB: default capacity - able to set optional value
leo_rpc leo_rpc
34

35.
DC-2 DC-3
Client Storage cluster Manager cluster
Monitors and Replicates each “RING” and “System Configuration”
Leo Storage Platform
Application(s)
DC-1
Stacked an object with a metadata
Compress it with LZ4
Replicated an object
Request to
the Target Region
leo_rpc
leo_rpc
leo_rpc
Multi Data Center Replication
Transferring stacked objects
Stacked objects
35

36.
leo_rpc
Multi Data Center Replication
Investigating stored objects
DC-2 DC-3
Client Storage cluster Manager cluster
leo_rpc
Monitor and Replicate each “RING” and “System Configuration”
Leo Storage Platform
Application(s)
Request to
the Target Region
DC-1
1) Receive metadata of stored objects
2) Compare them at the local cluster
3) Fix inconsistent objects
leo_rpc leo_rpc
36

37.
LeoFS v1.1
NFS Support
37

38.
LeoFS v1.1 - NFS Support
Offline - NFS / S3-API Online - REST / S3-API
PaaS / IaaS
BigData Analysis
Log / Event /
Sensor data
BigData
Analysis Platform
Document / Contents
Photo / Movie
Various Kind and
Huge Amount Data
Various Kind Data
Any Services
Log / Event /
Sensor data
38

39.
LeoFS Administration
at Rakuten
Presented by Paras Patel
An engineer of Rakuten DU
39

40.
LeoFS Administration at Rakuten
Storage Platform
File Sharing Service
End-User Servers
Portal Site
Photo Storage
Background Storage of OpenStack
40

41.
41
Rakuten Services are using LeoFS
41

42.
External Users
Project FiFo is an open-source Cloud Management and Orchestration
system for SmartOS virtualization environments.
The components of FiFo are written entirely in Erlang which gives the
suite excellent stability and fault recovery as it continues maturing to a
production quality release.
Project FiFo uses LeoFS packages for SmartOS in their repository.
42

43.
Storage Platform
43

44.
Storage Platform - Scaling the Storage Platform
(Movie)
Reduce Costs
High Reliability
Easy to Scale
S3-API
44

45.
Storage Platform - Scaling the Storage Platform
Using Various Services
Total Usage: 450TB/600TB
# of Files: 600Million
Daily Growth: 100GB
Daily Reqs: 13Million
E-Commerce
Review Photo
Blog
Insurance Calendar
share
Portal
Contents
Recruiting
B
Bookmark
Storage
Platform
(Movie)
45

46.
Monitor
GUI Console
Storage Platform - System Layout
( Erlang RPC)
( Erlang RPC)
( TCP/IP,SNMP )
Gateway x 4 Storage x 14
Manager x 2
Requests from
Web Applications / Browsers
w/HTTP over S3-API
Load Balancer / Cache Servers
Total disk space: 600TB
Number of Files: 600Million
Access Stats:
800Mbps (MAX)
400Mbps (AVG)
46

47.
Monitor
GUI Console
Ganglia Agent
( Erlang RPC)
( Erlang RPC)
( TCP/IP,SNMP )
Gateway x 4 Storage x 14
Manager x 2
Storage Platform - Monitor
Send Mail Alert
Status Collection (Ganglia)
Status Check (Nagios)
Port + Threshold Check
47

48.
Storage Platform - Spreading Globally
Covering All Services
with Multi DC Replication
48

49.
+
= +
https://owncloud.com/ http://roma-kvs.org/
49

50.
File Sharing Service - Required Targets
+
Reduce Costs
Handle Confidential Files
Store Large Files
Scale Easily
50

51.
File Sharing Service - Usage
+
Share Docs and Videos with Group Companies
Over 20 Companies, Over 10 Countries
Over 10,000 Users, Over 4,000 Teams
51

52.
LDAP
Monitor
Manager x 2
GUI Console
( Erlang RPC)
( Erlang RPC)
( TCP/IP,SNMP )
Authenticate Users
Manage
Configurations
Manage
Login Session
(KVS)
File Sharing Service - System Layout
Web GUI File Browser
52

53.
LeoFS Continuous Automated Testing
Continuous Integration with Jenkins
AWS SDK for PHP
AWS SDK for Ruby
AWS SDK for Java
Boto(Python)
erlcloud(Erlang)
s3cmd
s3fuse
LeoFS Client Test: https://github.com/leo-project/leofs_client_tests
53

54.
File Sharing Service - Future Plans
Cover 25 Countries/Regions
Over 20,000 Users
+
54

55.
Enhancing the Services
and Empowering the Users
through the Cloud Storage
55

56.
LeoFS v2.0
Beyond Storage System,
Toward Cloud Storage
56

57.
LeoFS v2.0 - Cloud Storage
Always keeps best condition of LeoFS
Autonomic Operation
Automatic Rebalance
Automatic Data Compaction
QoS
Statistics of internal LeoFS
Watchdog
Request / Traffic Control
57

58.
LeoFS v2.0 - Cloud Storage
Hybrid Storage
Centralizes Huge Amount and Various Kind data in LeoFS
Supports Online and Offline Access
PaaS / IaaS
BigData Analysis
Log / Event /
Sensor data
BigData
Analysis Platform
Document / Contents
Photo / Movie
Various Kind and
Huge Amount Data
Various Kind Data
Any Services
Log / Event /
Sensor data
58

59.
LeoFS v2.0 - Cloud Storage
PaaS / IaaS
BigData Analysis
Log / Event /
Sensor data
BigData
Analysis Platform
Document / Contents
Photo / Movie
Various Kind and
Huge Amount Data
Various Kind Data
Any Services
Log / Event /
Sensor data
REST-API (JSON)
Operate LeoFS
Notify a message of over # of req threshold
NewDB Proj
for LeoFS Insight
+
Retrieve metrics and stats from NewDB Proj's Agents
- Autonomic operation
- NFS Support
- Erasure Code
- NewDB Integration
- OpenStack Integration
and more...
59

60.
Introducing
NewDB Project
60

61.
NewDB Proj - Background
We need to get Value in Data in realtime
Typical System
Transaction Analysis Cache
NewDB Proj is a Hybrid DB
Handle Atomic Data
and Analyze Data in realtime
Typical Systems still realise
Semi-realtime Data Analysis
+
Database Statistics / ML
NewDB Proj
61

62.
NewDB Proj - Concept
Analyzing Data in Realtime
Retrieving Value in Exponential Data
Ad-hoc Query
Pluggable QL
ODBC/JDBC, REST, CLI,...
NewDB Proj
ML-lib
Integration
Pluggable ML/Statistics
Mechanism
High Realiability High Scalability
62

63.
Analyzing Data
Handle Event, Sensor
and Metrics Data
Online
Transaction Statistics, ML
Clustering Data
Finding Patterns in Data
Calculate and Generate Data
Then Grow Value of Data
Pluggable QL
Ease of Operation
and The QL Express Intuition Directly
NewDB Proj - Concept
SQL
Snapshot, Backup
Restore
Visualizing Data
63

64.
Early 2015
64

65.
Set Sail for the new Leo land
LeoProject
Website: leo-project.net
Twitter: @LeoFastStorage
65