Storage is one of the most important part of a data center, the complexity to design, build and delivering 24/forever availability service continues to increase every year. For these problems one of the best solution is a distributed filesystem (DFS) This talk describes the basic architectures of DFS and comparison among different free software solutions in order to show what makes DFS suitable for large-scale distributed environments. We explain how to use, to deploy, advantages and disadvantages, performance and layout on each solutions. We also introduce some Case Studies on implementations based on openAFS, GlusterFS and Hadoop finalized to build your own Cloud Storage.

1. Fabrizio Manfred Furuholmen!
Use Distributed File system as a Storage Tier !

2. 6/23/10!
2!
Agenda
 Introduction
 Next Generation Data Center
 Distributed File system
 Distributed File system
 OpenAFS
 GlusterFS
 HDFS
 Ceph
 Case Studies
 Conclusion

3. 6/23/10!
3!
Class Exam
 What do you know about DFS ?
 How can you create a Petabyte
storage ?
 How can you make a centralized
system log ?
 How can you allocate space for your
user or system, when you have a
thousands of users/systems ?
 How can you retrieve data from
everywhere ?

4. 6/23/10!
4!
Introduction
Next Generation Data Center: the “FABRIC”
Key categories:
 Continuous data protection and disaster
recovery
 File and block data migration across
heterogeneous environments
 Server and storage virtualization
 Encryption for data in-flight and at-rest
In other words: Cloud data center

5. 6/23/10!
5!
Introduction
Storage Tier in the “FABRIC”
 High Performance
 Scalability
 Simplified Management
 Security
 High Availability
Solutions
 Storage Area Network
 Network Attached Storage
 Distributed file system

6. 6/23/10!
6!
Introduction
What is a Distributed File system ?
“A distributed file system takes advantage of the
interconnected nature of the network by storing
files on more than one computer in the network
and making them accessible to all of them..”

7. What do you expected from a distributed file system ?
• Uniform Access: file names global support
• Security: to provide a global authentication/authorization
• Reliability: the elimination of each single point of failure
• Availability: administrators perform routine maintenance while the file
server is in operation, without disrupting the user’s routines
• Scalability: Handle terabytes of data
• Standard conformance: some IEEE POSIX file system semantics standard
• Performance: high performance
Introduction
7!

8. Part II
Implementations
8!
How many DFS do you know ?

9. 6/23/10!
9!
OpenAFS: introduction
Key ideas:
 Make clients do work whenever possible.
 Cache whenever possible.
 Exploit file usage properties. Understand them. One-third of Unix
files are temporary.
 Minimize system-wide knowledge and change. Do not hardwire
locations.
 Trust the fewest possible entities. Do not trust workstations.
 Batch if possible to group operations.
is the open source implementation of
Andrew File system of IBM

10. 6/23/10!
10!
OpenAFS: design

11. • Cell is collection of file servers and
workstation
• The directories under /afs are cells,
unique tree
• Fileserver contains volumes
Cell
• Volumes are "containers" or sets of
related files and directories
• Have size limit
• 3 type rw, ro, backup
Volumes
• Access to a volume is provided through
a mount point
• A mount point is just like a static
directory
Mount Point Directory
OpenAFS: components
Server A
Server A+B
Server C
11!

12. OpenAFS: performances
OpenAFS OpenAFS OSD 2 Servers

13. 6/23/10!
13!
OpenAFS: features
 Uniform name space: same path on all
workstations
 Security: base to krb4/krb5, extended ACL,
traffic encryption
 Reliability: read-only replication, HA
database, read/write replica in OSD version
 Availability: maintenance tasks without
stopping the service
 Scalability: server aggregation
 Administration: administration delegation
 Performance: client side disk base persistent
cache, big rate client per Server

14. • Internal usage
• Storage: 450 TB (ro)+ 15 TB (rw)
• Client: 22.000
Morgan Stanley IT
• Online picture album
• Storage: 265TB ( planned growth to 425TB in twelve months)
• Volumes: 800,000.
• Files: 200 000 000.
Pictage, Inc
• Internet Shared folder
• Storage: 500TB
• Server: 200 Storage server
• 300 App server
Embian
• Internal usage 210TB
RZH
openAFS: who uses it ?
14!

15. Good
• Wide Area Network
• Heterogeneous System
• Read operation > write operation
• Large number of clients/systems
• Usage directly by end-users
• Federation
Bad
• Locking
• Database
• Unicode
• Large File
• Some limitations on ..
OpenAFS: good for ...
15!

16. 6/23/10!
16!
GlusterFS
“Gluster can manage data in a
single global namespace on
commodity hardware..”
Keys:
 Lower Storage Cost—Open source software runs on commodity
hardware
 Scalability—Linearly scales to hundreds of Petabytes
 Performance—No metadata server means no bottlenecks
 High Availability—Data mirroring and real time self-healing
 Virtual Storage for Virtual Servers—Simplifies storage and keeps VMs
always-on
 Simplicity—Complete web based management suite

17. 6/23/10!
17!
GlusterFS: design

18. 6/23/10!
18!
GlusterFS: components
• Volume is the basic element for data
export
• The volumes can be stacked for
extension
Volume
• Specific options (features) can be
enabled for each volume (cache, pre
fetch, etc.)
• Simple creation for custom extensions
with api interface
Capabilities
• Access to a volume is provided through
services like tcp, unix socket,
infiniband
Services
volume posix1!
type storage/posix!
option directory /home/export1!
end-volume!
volume brick1!
type features/posix-locks!
option mandatory!
subvolumes posix1!
end-volume!
volume server!
type protocol/server!
option transport-type tcp !
option transport.socket.listen-port 6996!
subvolumes brick1!
option auth.addr.brick1.allow * !
end-volume!

19. 6/23/10!
19!
Gluster: components

20. 6/23/10!
20!
Gluster: performance

21. 6/23/10!
21!
Gluster: carateristics
 Uniform name space: same path on all
workstation
 Reliability: read-1 replication, asynchronous
replication for disaster recovery
 Availability: No system downtime for
maintenance (better in the next release)
 Scalability: Truly linear scalability
 Administration: Self Healing, Centralized logging
and reporting, Appliance version
 Performance: Stripe files across dozens of
storage blocks, Automatic load balancing, per
volume i/o tuning

22. Gluster: who uses it ?
 Avail TVN (USA)
400TB for Video on demand, video
storage
 Fido Film (Sweden)
visual FX and Animation studio
 University of Minnesota (USA)
142TB Supercomputing
 Partners Healthcare (USA)
336TB Integrated health system
 Origo (Switzerland)
open source software development
and collaboration platform
22!

23. Good
• Large amount of data
• Access with different protocols
• Directly access from applications
(api layer)
• Disaster recover (better in the
next release)
• SAN replacement, vm storage
Bad
• User-space
• Low granularity in security setting
• High volumes of operations on
same file
Gluster: good for ...
23!

24. 6/23/10!
24!
Implementations
Implementations
Old way
 Metadata and data in the same place
 Single stream per file
New way
 Multiple streams are parallel channels
through which data can flow
 Files are striped across a set of nodes in
order to facilitate parallel access
 OSD Separation of file metadata
management (MDS) from the storage of
file data

25. 6/23/10!
25!
HDFS: Hadoop
HDFS is part of the Apache
Hadoop project which
develops open-source software
for reliable, scalable,
distributed computing.
Hadoop was inspired by Google’s
MapReduce and Google File
system

26. 6/23/10!
26!
HDFS: Google File System
“ Design of a file systems for a different environment
where assumptions of a general purpose file system
do not hold—interesting to see how new assumptions
lead to a different type of system…”
Key ideas:
 Component failures are the norm.
 Huge files (not just the occasional file)
 Append rather than overwrite is typical
 Co-design of application and file system API—specialization.
For example can have relaxed consistency.

27. Map!
• Split and mapped in key-
value pairs!
Combine!
• For efficiency reasons, the
combiner works directly to map
operation outputs .!
Reduce!
• The files are then merged,
sorted and reduced!
“Moving Computation is Cheaper than Moving Data”
HDFS: MapReduce
27!

28. Goals!
Scalable: can reliably store and
process petabytes.!
Economical: It distributes the data and
processing across clusters of
commonly available computers. !
Efficient: can process data in parallel
on the nodes where the data is
located.!
Reliable: automatically maintains
multiple copies of data and
automatically redeploys computing
tasks based on failures.!
HDFS: goals
28!

29. HDFS: design
29!

30. • An HDFS cluster consists of a single
NameNode
• It is a master server that manages
the file system namespace and
regulates access to files by clients.
Namenode
• Datanode manage storage attached
to the system it run on
• Applay the map rule of MapReduce
Datanodes
• File is split into one or more blocks
and these blocks are stored in a set
of DataNodes
Blocks
HDFS: components
30!

31. 6/23/10!
31!
HDFS: features
 Uniform name space: same path on all
workstations
 Reliability: rw replication, re-balancing, copy
in different locations
 Availability: hot deploy
 Scalability: server aggregation
 Administration: HOD
 Performance: “grid” computation, parallel
transfer

32. HDFS: who uses it ?
Major players
Yahoo!
A9.com
AOL
Booz Allen Hamilton
EHarmony
Facebook
Freebase
Fox Interactive Media
IBM
ImageShack
ISI
Joost
Last.fm
LinkedIn
Metaweb
Meebo
Ning
Powerset (now part of Microsoft)
Proteus Technologies
The New York Times
Rackspace
Veoh
Twitter
…
32!

33. Good
• Task distribution (Basic GRID
infrastructure)
• Distribution of content (High
throughput of data access )
• Archiving
• Etherogenous envirorment
Bad
• Not General purpose File system
• Not Posix Compliant
• Low granularity in security setting
• Java
HDFS: good for ...
33!

34. Ceph
“Ceph is designed to handle workloads
in which tens thousands of clients or
more simultaneously access the same
file or write to the same directory–
usage scenarios that bring typical
enterprise storage systems to their
knees.”
Keys:
 Seamless scaling — The file system can be seamlessly expanded by simply
adding storage nodes (OSDs). However, unlike most existing file systems, Ceph
proactively migrates data onto new devices in order to maintain a balanced
distribution of data.
 Strong reliability and fast recovery — All data is replicated across multiple
OSDs. If any OSD fails, data is automatically re-replicated to other devices.
 Adaptive MDS — The Ceph metadata server (MDS) is designed to dynamically
adapt its behavior to the current workload.
34!

35. • Client
• Metadata
Cluster
• Object
Storage
Cluster
OSD
Ceph: design
35!

36. • Metadata Storage
• Dynamic Subtree Partitioning
• Traffic Control
Dynamic Distributed Metadata
• Data Distribution
• Replication
• Data Safety
• Failure Detection
• Recovery and Cluster Updates
Reliable Autonomic Distributed Object
Storage
Ceph: features
36!

37. Pseudo-random data distribution function (CRUSH)!
Reliable object storage service (RADOS)!
Extent B-tree object File System (today btrfs) !
37!
Ceph: features

38. Splay Replication
• Only after it has been safely committed to disk is a final commit
notification sent to the client.
Ceph: features
38!

39. Good
• Scientific application, High
throughput of data access
• Heavy Read / Write operations
• It is the most advance distributed
file system
Bad
• Young (Linux 2.6.34)
• Linux only
• Complex
Ceph: good for …
39!

40. Lustre
Cloudstore
(kosmos)!
XtreemFS! Tahoe-LAFS!
PNFS!
PVFS! MooseFS!
…!
Search
Wikipedia..!
Others
40!

41. Part III
Case Studies
41!

42. 6/23/10!
42!
Class Exam
 What can DFS do for you ?
 How can you create a Petabyte
storage ?
 How can you make a centralized
system log ?
 How can you allocate space for your
user or system, when you have a
thousands of users/systems ?
 How can you retrieve data from
everywhere ?

43. • Share Documents across a wide
network area
• Share home folder across different
Terminal servers
Problem
• OpenAFS
• Samba
Solution
• Single ID, Kerberos/ldap
• Single file system
Results
• 800 users
• 15 branch offices
• File sharing /home dir
Usage
File sharing
43!

44. • Big Storage on a little budget
Problem
• Gluster
Solution
• High Availability data storage
• Low price
Results
• 100 TB image archive
• Multimedia content for web site
Usage
Web Service
44!

45. • Data from everywhere
• Disaster Recover
Problems
• myS3
• Hadoop / OpenAFS
Solution
• High Availability
• Access through HTTP protocol (REST
Interface)
• Disaster Recovery
Results
• Users backup
• Application backend
• 200 Users
• 6 TB
Usage
Internet Disk: myS3
45!

46. • Log concentrator
Problem
• Hadoop cluster
• Syslog-NG
Solution
• High availability
• Fast search
• “Storage without limits”
Results
• Security audit and access control
Usage
Log concentrator
46!

47. • Low cost VM storage
• VM self provisioning
Problems
• GlusterFS
• openAFS
• Custom provisioning
Solution
• Auto provisioning
• Low cost
• Flexible solution
Rresults
• Development env
• Production env
Usage
Private cloud

48. 6/23/10!
48!
Conclusion: problems
 Failure
For 10 PB of storage, you will have an
average of 22 consumer-grade SATA drives
failing per day.
 Read/write time
Each of the 2TB drives takes approximately
best case 24,390 seconds to be read and
written over the network.
 Data Replication
Data replication is the number of the disk
drives, plus difference.
Do you have enough bandwidth ? !

49. Environment Analysis!
• No true Generic DFS!
• Not simple move 800TB btw different solutions!
Dimension !
• Start with the right size!
• Servers number is related to speed needed and number of clients !
• Network for Replication !
Divide system in Class of Service!
• Different disk Type !
• Different Computer Type!
System Management!
• Monitoring Tools!
• System/Software Deploy Tools!
Conclusion
49!

50. 6/23/10!
50!
Conclusion: next step

51. OpenAFS!
• www.openafs.org!
• www.beolink.org!
Gluster!
• www.gluster.org!
Hadoop!
• Hadoop.apache.org!
• Isabel Drost!
Ceph!
• ceph.newdream.n
et!
• Publication!
• Mailing list!
Links
51!

52. I look forward to meeting you…
XVII European AFS meeting 2010
PILSEN - CZECH REPUBLIC
September 13-15
Who should attend:
 Everyone interested in deploying a globally accessible
file system
 Everyone interested in learning more about real
world usage of Kerberos authentication in single
realm and federated single sign-on environments
 Everyone who wants to share their knowledge and
experience with other members of the AFS and
Kerberos communities
 Everyone who wants to find out the latest
developments affecting AFS and Kerberos
More Info: http://afs2010.civ.zcu.cz/
6/23/10!
52!

53. Thank you!
manfred@zeropiu.com