> MogileFS Architecture
Philippe Julio – Principal Field Technologist
Sun Microsystems France
November, 2009

Data Management
Vision
« The Data are not created relevant,
they become so ! »
November, 2009 2

What Is MogileFS ?
MogileFS is an open source distributed
filesystem, flexible and high available on a
network of commodity hardware
MogileFS is an anagram for "OMG Files" and
was created for LiveJournal to handle the
storage, replication and retrieval of the large
amount of file uploads.
November, 2009 3

Data Management
Data-Driven on-Line Websites
• To run the apps : messages, posts, blog entries, video
clips, maps, web graph...
• To give the data context : friends networks, social
networks, collaborative filtering...
• To keep the applications running : web logs, system logs,
system metrics, database query logs...
November, 2009 4

Sun's Cloud Computing Strategy
Products Expertise
Q-layer and and
Products Expertise
Technologies andServices
and
Technologies Services
xVM
OpenOpen
Partners Community
Communities
November, 2009 5

Sun's Open Source Strategy
Developer Preference User Preference Value Proposition
●
More core developers • Free to use ●
Business Deployment
●
More deploying • More platform choice ●
Sun's target market
developers • More suppliers ●
Binary distribution
●
More partners • Larger user community ●
Pay for value
November, 2009 6

New Data Management Economics
Compute Trend Data Trend
New Analytics Emerge Semi-structured Data
(MapReduce...) (MogileFS, Bigtable, HDFS...)
Master/Slave
Semi-
Object Store structured
Architectural shift to the cloud Database
and HPC-style workloads ScaleDB, Big Table,
SimpleDB HBase Master/Master
Open source, general
purpose datawarehouse
Distributed FS Federated/
Proprietary, dedicated Sharded
datawarehouse
Unstructured Structured
Data Data
OLTP is the
datawarehouse
November, 2009 7

Changing Software Economics
1998 2008 2009
November, 2009 8

Who Used MogileFS ?
• Danga Interactive project
• Six appart acquisition
220 million pages view / Month
5 billion pages view / Month
November, 2009 9

How to Interface your Application
with MogileFS ?
• Implement “Save As Cloud...” to Write to the cloud
• Implement “Open From Cloud...” to Read from the
cloud
• MogileFS 2.32
http://search.cpan.org/~bradfitz/mogilefs-server
• Documentation Guide
http://www.danga.com/mogilefs
• Develop your application with MogileFS :
http://mogilefs.pbworks.com/ObtainingMogileFS
November, 2009 10

OpenSolaris for MogileFS
Network AutoMagic Automated Install
Security
D-Light Image Packaging
Containers System
ZFS DTrace
Time Slider CIFS
Predictive Self Healing
Distribution Solaris Virtualization
Constructor Technology
Open Storage COMSTAR
November, 2009 11

MogileFS Architecture
Infrastructure as a Services
General Purpose Storage Servers
• Combine server with disk & networking
• Specialized software enables general purpose
systems designs to provide high performance data
services
Sun's Open Platform direction Data moves to the infrastructure
Sun Fire x4xxx
(Data Compute and Store)
Sun Sparc Enterprise T5xxx
(Data Compute and Store)
Sun Storage 7xxx
(Data Store)
November, 2009 12

MogileFS Architecture
Sizing
• Application Servers sizing
> Number of concurrent low users
> Number of concurrent medium users
> Number of concurrent high users
• MogileFS Servers sizing
> Useable Data Volume : Customer data volume (customer need)
> Number of replication blocks : 2 minimum (3 recommanded)
> 2 CPU quad-core
> 16 GB RAM minimum
> Work Data Volume (metadata, namespace...)
> Raw Data Volume (Useable Data Volume * Nb of replication blocks) + Work Data Volume
> Number of cluster nodes
Max (Number of replication blocks, (Number of Trackers + Number of Strorage Nodes))
• No RAID factor, No HBA port
November, 2009 13

MogileFS Architecture
x86 Components
Low Cost Server & Storage : Sun Fire x4xxx
Interface
Sun Virtualization Technologie : Solaris Containers
Performance
• NFS
• HTTP
• WebDAV
Sun Fire x4540
● 2 CPU Quad Core AMD
Sun Fire x4240 ● Up to 64 GB RAM
● 2 CPU Quad Core AMD 48 x 1TB SATA II 7200 RPM
Sun Fire x4140
●
● Up to 128GB RAM ● 4 RU
● 2 CPU Quad Core AMD ● Up to 4,67 TB Disks
● Up to 128 GB RAM ● 16 x 300 GB SAS 10000 RPM
● Up to 2,34 TB Disks ● 2 RU
● 8 x 300 GB SAS 10000 RPM
● 1 RU
Capacity
November, 2009 14

MogileFS Architecture
CMT Components
Low Cost Server : Sun Sparc Enterprise T5xxx
Interface
Sun Virtualization Technologies : Solaris Containers, LDoms
Performance
• NFS
• HTTP
Sun Enterprise T5240 • .WebDAV
● 2 CPU Octo Core UST2+
● Up to 256 GB RAM
Sun Enterprise T5440 ● Up to 4,68 TB Disks
● 2 CPU Octo Ccore UST2+ ● 16 x 300 GB SAS 10000 RPM
● Up to 512GB RAM ● 2 RU
Sun Blade T6340 ● Up to 1,17 TB Disks
● 2 CPU Octo Core UST2+
● 4 x 300 GB SAS 10000 RPM
● Up to 256 GB RAM
● 4 RU
● Up to 600 GB Disk
● 2 x 300 GB 10000 RPM
● 10 RU
Capacity
November, 2009 15

MogileFS Architecture
Open Storage Components
Low Cost Storage : Sun Storage 7xxx OpenStorage
Interfaces
Performance
• NFS
• WebDAV
• HTTP
• D-Trace
Alalytics
Sun Storage 7410
● 16GB, 64GB and 128GB
Sun Storage 7210 RAM options
● 32GB and 64GB RAM options ● Up to 288TB
Sun Storage 7110 ● Up to 142TB ● Hybrid Storage Pool I/O
● 8GB RAM ● Hybrid Storage Pool I/O Acceleration
● 14 x 300 GB SAS 10K Acceleration ● Read/Write Flash/SSD options
RPM Drives ● Write Flash/SSD options for higher performance
● Up to 4, 2TB for higher performance
Capacity
November, 2009 16

MogileFS Architecture
Key Enablers
• A scalable, Fault tolerant, High performance distributed file
system
• No Single Point of Failure
Tracker Storage Node
• Automatic file replication (3 replications recommanded)
• Better than RAID
Host1 Host4
• Flat NameSpace
• Share-Nothing
• No RAID required
Tracker Storage Node
• Local filesystem agnostic
• Tracker client transfert (mogilefsd) - Replication -- Deletion
- Query - Reaper - Monitor Host2 Host5
• Files are broken up and spread over the
Storage Node (mogstored) HTTP and WebDAV server
• Database MySQL stores the MogileFS metadata (the Database Storage Node
namespace, and which files are where)
• Client Library : Ruby, Perl, Java, Python, PHP…
Host3 Host6
November, 2009 17

MogileFS Architecture
Overview
database
Client Library
Tracker Tracker
Storage Node Storage Node
November, 2009 18

MogileFS Architecture
World Wide IP Cluster HA
Storage Node
Tracker Storage Node
Storage Node Tracker Storage Node Storage Node
Storage Node
Tracker
Database
Storage Node
Hybrid Server High Avalability High Scalability
(compute and storage) with load balancing and by adding
for all Cluster nodes Cluster Storage Nodes
November, 2009 19

MogileFS Architecture
Software Components
• MogileFS • Memcached • Perlbal
> Is an open source > Is a high-performance, > Is Perl-based reverse proxy
distributed filesystem distributed memory object load balancer and web server
> 3 components : (storage caching system, generic in > Is a single-threaded event-
nodes, trackers, and the nature, but intended for use in based server supporting
database speeding up dynamic web HTTP load balancing, web
applications by alleviating serving, and a mix of the two
> Local disks on storage database load
nodes can be formatted > dropped the database load to
> The management interface
with your filesystem of provides extremely detailed
choice (ZFS, NFS...) almost nothing, yielding faster and powerful statistics in
page load times for users, addition to runtime
better resource utilization, and configuration
faster access to the
databases on a memcache > Supports the concept of
miss having per-service (and
global) plugins that can
override many parts of request
handling and behavior.
November, 2009 20

MogileFS Architecture
High Availability
MogileFS Architecture
Database Tracker Storage Storage Storage
Single Point Single Point
Of Failure 0 Of Failure 1 2 3 4
(SPOF) (SPOF)
MogileFS High Availability Architecture
Database Tracker Tracker Tracker Storage Storage Storage Storage
Active Passive
Cluster 0 0' Cluster 1 2 3 4 5 6 7
Node Node
(Metadata)
2 Trackers nodes for HA Cluster 4 Storage Nodes
November, 2009
and more than 2 for Load Balancing 21

MogileFS Architecture
High Availability
2 x HD 300 GB 2 x HD 300 GB
Active Database node Passive Database Node
+ Metadata + Metadata
Solaris Cluster
Sun StorageTek Array
Disks Mirroring
RAID1
12 Disks SAS 146 GB
November, 2009 22

MogileFS Architecture
x86 Reference Architecture
Cloud Storage Model
Sun Fire x4140
Application Servers ● 2 CPU Quad Core AMD
● 32 GB RAM
SF x4140 SF x4140 ● 2 x 300 GB SAS 10000 RPM
SF x4140 SF x4140 SF x4540 Sun Fire 4540
● 2 CPU Quad Core AMD
● 32 GB RAM
Database Tracker ● 48 x 1TB SATA II 7200 RPM
Storage Node
Interfaces
• WebDAV
OpenSolaris on each server • NFS
MogileFS on each cluster node • HTTP
November, 2009 23

MogileFS Architecture
CMT Reference Architecture
Cloud Storage Model
Sun Enterprise T5240
Application Servers ● 2 CPU Octo Core UST2+
● 32 GB RAM
SE T5240 SE T5240 ● 2 x 300 GB SAS 10000 RPM
SE T5240 SE T5240 SE T5240 Sun Enterprise T5240
● 2 CPU Oc*to Core UST2+
Database Tracker Storage Node ● 32 GB RAM
● 16 x 300 GB 10000 RPM
Interfaces
• WebDAV
OpenSolaris on each server • NFS
MogileFS on each cluster node • HTTP
November, 2009 24

MogileFS Architecture
Open Storage Reference Architecture
SF x4140
Cloud Storage Model
Database
Sun Fire x4140 Tracker
● 2 CPU Quad Core
● 32 GB RAM Tracker
● 4 x 300 GB SAS
10000 RPM Tracker OpenStorage
Storage Node
Switch
Storage Node
Storage Node Sun Storage 7410
● 64 GB RAM
Storage Node ● 288 TB Interfaces
Hybrid Storage Pool I/O
WebDAV
●
Storage Node Acceleration •
● Read/Write Flash/SSD • NFS
Storage Node options for higher
performance
• HTTP
Storage Node
OpenSolaris on each server
MogileFS on each cluster node
Performances analysis with D-Trace Analytics on Sun Storage 7xxx
November, 2009 25

MogileFS Architecture
Pricing Model - Example
• Utility pay-as-you-go pricing model,
competitive to market
• Based on capacity
> Volume of data used
> Volume of data transfer
> ...
• Easy sign-up and self provisioning
November, 2009 26

MogileFS Infrastructure
Value Proposition
Manage terabytes of • Mean processing time •
TECHNOLOGY / METHODOLOGY
CMT, AMD and INTEL Processors
KEY PERFORMANCE INDICATORS
CUSTOMER REFERENCE
• Volume of data used
storage with little effort • Multi OS : Linux, Solaris 10
• Volume of data transfer
Objective : manage terabytes of storage • Open Solaris
• Number of requests
with little effort • Sun Cloud Storage
• Number of Storage nodes
Solution : more hundred of servers • Open Storage : Sun Storage 7000 Unified Storage
• Number of Trackers
Customer Benefit : this client was able
System
• Return On Investment
to utilize hundreds of machines • Sun Fire x4540 (Hybrid Server with 48TB)
concurrently. No captial expense for • Total Cost of Ownership
equipment. Ability to deploy in hours not • Infratsructure SwaP ratio • MogileFS is a distributed applications with high
in months density of data
• Use rate of equipment environment
• Time to deploy a new service • MySQL is the database for MogileFS metadata
storage
• Time to Market
• Memcached is a high-performance, distributed
memory object caching system, generic in nature,
but intended for use in speeding up dynamic web
applications by alleviating database load
• Perlbal is a reverse proxy for load balancer and web
server
• Solaris Containers
• Sun xVM Ops Center for patching and provisonning
• Cloud Computing Workshop
SERVICES
• Solaris Cluster
• Sun Cloud Services (compute, storage)
• Product Deployment Services • Solaris ZFS
• Sun Learning Services
• Sun Managed Services
• Sun Support Services
• Sun Global Financial Services Operation
November, 2009 27

Philippe Julio
philippe.julio@sun.com
http://blogs.sun.com/philippejulio