Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

Like this? Share it with your network

Share

GlusterFS Community Preso

  • 2,881 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,881
On Slideshare
2,879
From Embeds
2
Number of Embeds
2

Actions

Shares
Downloads
58
Comments
0
Likes
2

Embeds 2

https://si0.twimg.com 1
http://www.linkedin.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • TONY– Softball Questions 1.       Fred from Kansas asked: How does Cloud Computing differ from Out Sourcing? - Tony 2.       Alan from Missouri asked:  Does Gluster support point in time recovery? – Jack 3.       David from South Africa asked: Are there feasible 3rd party cloud storage options available to compliment cloud compute systems? – Tony 4.       Does Gluster have a viable business model?  How do they make money? – Tony 5.       What was your biggest storage challenge before implementing Gluster and how did Gluster help solve this issue? – Brent & Barry 6.       How many customers are using Gluster in production today? – Jack MIKE Folks, we’re at that time and want to thank each of you for joining us today and hope that you found today’s webinar Case Studies: Deploying Open Source Storage Clouds of value and you’ll want to join us for future events. You’ll be receiving a short survey at the end of this webinar, and we’d sure appreciate your filling it out as we value your time and feedback. Additionally, for those questions we couldn’t answer on air, we’ll have Gluster work with each subject matter expert to reply to your questions and a copy of today’s recorded webcast will be make available as an On Demand Viewing, and you’ll each be receiving an email with a link to access this recording by early next week. Special thanks to our guests Tony Asaro, Barry Jaspan and Brent Richter for joining us today with us, as well as Gluster for sponsoring today. And don’t forget by completing the short survey that will appear on your screen momentarily you’ll have a chance to win a Jawbone ICON™ headset . We will be drawing a name at random and notifying the winner by email on Friday. And again as your moderator, Mike Agron saying thank you again and have a great rest of the week, good bye for now! End Webinar, and everyone dial in for the debrief.

Transcript

  • 1. The Future of Gluster.org John Mark Walker Gluster Community Guy Red Hat, Inc. December 6, 2011
  • 2. History
    • The Beginning of GlusterFS
      • Software only
      • 3. Open Source, community-oriented
      • 4. A vision of commoditized hardware
    • The Middle Ages
      • Less community, focus on revenue
      • 5. No published roadmap or community process
    • Pre-Red Hat
      • Hired a community manager (moi)
      • 6. Oh yeah, the community thing
  • 7. A Brief History of Storage
    • Enter the SAN
      • Complexity increased over time
        • Block devices virtualized
        • 8. Underlying OS and filesystem
        • 9. Actual disks sliced into array of virtual bricks
    Disk 1 Disk 2 Disk 3 COMPELLENT / EMC / HDS / IBM / NTAP Virtual Disks
  • 10. A Brief History of Storage
    • NAS Marketecture
      • Look familiar?
    Disk 1 Disk 2 Disk 3 Operating System Exported Folders File System (i.e. WAFL, DART, etc) +
  • 11. A Brief History of Storage
    • So what happened? Economics
      • Entire business models enabled by new computing
      • 12. Pool together cheap, commodity parts to achieve scale
        • The Google approach
      • Facebook, Twitter, Google not possible with traditional, proprietary approach
      • 13. New services emerging daily enabled by economics of new computing
  • 14. A Brief History of Storage
    • Except for one thing…
    Storage. Still. Sucks.
  • 15. A Brief History of Storage
    • The Lustre / Metadata Server Way
      • There’s gotta be a better way to distribute storage load, while avoiding EMC and NetApp
      • 16. Let’s create distributed file storage
        • Cheaply and open source
        • 17. POSIX-compliant for legacy apps
        • 18. Metadata server
    Storage node 1 Storage node 2 Storage node 3 Metadata Server
  • 19. A Brief History of Storage
    • The Lustre / Metadata Server Way
      • Pros:
        • Circumvents EMC and NetApp
        • 20. Scaling with commodity hardware + software
      • Cons:
        • Metadata server must run in memory
          • Hard limits to scaling
  • 21. Aside: Contrast With Lustre
    • Need one MGS, one MDT, one or more OSTs
      • Separate LUNs for each
      • 22. Custom formats, many options/dependencies
    • Start MGS + MDT + OSTs
      • Careful about ordering
    • Want high availability? Set up your own heartbeat/failover.
  • 23. A Brief History of Storage
    • Simplifying Storage
      • Proprietary storage not a valid option for the Googles and Facebooks of the world – what to do?
      • 24. Simplification – let’s do it the Googly way.
    JBODs Buckets
  • 25. A Brief History of Storage
    • The Googly Way
      • Pros:
        • Greatly simplified – only “puts” and “gets”
        • 26. Takes advantage of advances in computing
        • 27. Much cheaper storage arrays
      • Cons:
        • Do you have an army of web developers and engineers?
        • 28. Do you enjoy reimplementing the equivalent of ls, rm, mv, and cp?
  • 29. A Brief History of Storage
    • The Amazon Way
      • Much like Google
        • S3 – “Simple Storage Service”
        • 30. Object-based
        • 31. HTTP
        • 32. Puts and gets
      • Brought object storage to the masses
        • Now there was a simple, relatively cheap way to store data in the cloud
        • 33. Open source implementation in the form of Swift
  • 34. A Brief History of Storage
    • The HDFS Way
      • Created for Hadoop
        • Simplified, single-purpose storage
        • 35. Scales via “name node”, ie. Metadata server
      • Pros:
        • Can scale
        • 36. Doesn’t need POSIX or legacy app support
      • Cons:
        • No legacy app support
        • 37. Not general purpose
  • 38. A Brief History of Storage
    • Another Way: Virtualization
      • Enabled by massive growth in compute power
      • 39. New VM’s created and destroyed in seconds
      • 40. These VM’s attach virtual bricks stored on the base filesystem
      • 41. Enabled commodity distributed storage
    Disk Disk Disk Filesystem + OS / Hypervisor VMs Virtual Bricks
  • 42. A Brief History of Storage
    • Remember SANs?
      • Virtualization commoditized block storage
      • 43. Pros:
        • Virtual disks could be housed on cheap, commodity hardware, hosted on open source file systems and OS
        • 44. Virtual disks could be accessed via variety of methods
          • Services running on VM
          • 45. Mounting hypervisor filesystem elsewhere
      • Cons:
        • Scaling issues beyond the hypervisor
        • 46. Vmware stack relatively expensive (compared to rest of commoditized computing)
  • 47. The General Problem
    • To solve the expensive storage problem, all solutions are either
      • Not Scalable, or…
      • 48. Single purpose, or…
      • 49. Doesn’t support legacy apps, or…
      • 50. Do them, but not very well
  • 51. The General Problem
    • Gluster’s challenge:
      • Create a storage system that was…
        • Scalable
          • Vertically + horizontally
        • Seamlessly integrated in the data center
        • 52. Future-proof
    • Gluster’s solution: GlusterFS
  • 56. GlusterFS
    • Creating a file system in user space
      • Utilizes fuse module
        • Kernel goes through fuse, which hands off to glusterd
    Linux kernel Fuse Ext4 glusterd … … Applications
  • 57. It's a Filesystem!
    • Open, close, read, write
      • ...rename, stat, fsync, links, etc. ad nauseam
    • No new data models, query languages
    • 58. No change to applications
    • 59. Thousands of tools
    • 60. Try storing kernel sources in a NoSQL DB
      • ...then type “make”
  • 61. No Single Point Of Failure SPOF
  • 62. No Centralized Metadata Client A Client B Client C Server X Data Metadata Server Y Data Metadata Server Z Data Metadata
  • 63. Client Driven Behavior
    • Move effort to most numerous component
    • 64. Servers are dumb “bricks”
    YES NO Client Server 1 Server 2 Client Server 1 Server 2
  • 65. Leverage Local Filesystems
    • All user data is in server filesystems
      • Directories, files, xattrs
      • 66. Reconstruct using standard filesystem tools
    • No special-purpose local filesystem
    • 67. No database
    • 68. No private/undocumented formats
  • 69. Translator Interface
    • Add/remove layers
    • 70. Reorder layers
    • 71. Move layers between client and server
    • 72. Implement new layers
      • e.g. encryption
    • Replace old layers
      • e.g. replication
    FUSE Interface Layer Performance Layer Distribution Layer Replication Layer Protocol Layer Local Filesystem Later
  • 73. Setting Up a Cluster
    • yum install glusterfs-*
    • 74. /etc/init.d/glusterd start
    • 75. gluster peer probe other_server
  • 76. Creating and Mounting
    • gluster volume create myvol server1:/local/fs server2:/local/fs
    • 77. gluster volume start myvol
    • 78. mount -t glusterfs server1:myvol /mount/point
  • 79. Other Tricks
    • Rebalance
    • 80. Remove or replace bricks
    • 81. gluster volume profile
    • 82. gluster volume top
  • 83. GlusterFS 3.3: Unified File and Object Storage
    • Traditional SAN / NAS devices support either file or block storage.
      • NFS & CIFS
    • New storage methodologies are implementing REST for object storage.
      • HTTP(s) / Get / Put
      • 84. OS agnostic
    • Demand for unifying the storage infrastructure increasing in the Block / File arena
    • 85. Goal: treat files as objects and volumes as buckets
  • 86. The Software Approach
    • Network Attached Storage (NAS)
      • NFS / CIFS / GlusterFS
      • 87. POSIX compliant
      • 88. Access files within objects
    • Window Access
      • Improves Windows performance
      • 89. Uses HTTP, not slower CIFS
      • 90. We will still support SAMBA
    • Object Storage
    • Standards based
      • Amazon S3 ReSTFul interface compatible
      • 96. Access data as objects and a NAS interface to access files (NFS, CIFS, GlusterFS)
    High performance storage across heterogeneous server environments Object Info >>>> >>>> Objects Files
  • 97. The Gluster Connector for *Stack SWIFT
    • GlusterFS used as VM storage system
      • Pause and re-start VM’s, even on another hypervisor
      • 98. HA and DR for VM’s
      • 99. Faster VM deployment
      • 100. V-motion –like capability
    OpenStack Imaging Services Unified File & Object Storage … Compute API Layer Mobile Apps. Web Clients. Enterprise Software Ecosystem OpenStack Today
  • 101. GlusterFS Compatibility for Apache Hadoop (HDFS)
    • An enhancement to GlusterFS that provides a new file system option for Apache Hadoop
      • Proven scale-out storage solution provides simultaneous file and object access within Hadoop
      • 102. Introduces a 4th storage option for Hadoop (HDFS, local disk, Kosmos)
    • Included in GlusterFS 3.3 beta 2
    • 103. Requirements driven by community and customer requests
      • “ Provide more flexibility than the 64MB recommended block size imposed by HDFS”
      • 104. “ Eliminate the centralized metadata server” (name node)
      • 105. “ Give us NAS connectivity” (via POSIX compliance)
      • 106. Let me update a single file
    • Benefits
      • Out of the box compatibility with MapReduce applications, no rewrite required
      • 107. Enables organizations to unify data storage
      • 108. Flexible and powerful, it simplifies access and management of data
  • 109. Uses for GlusterFS and Hadoop
    • Simplify and unify storage deployments
      • Centralized data store providing access to more applications
    • Provide users with file level access to data
      • Users can easily brose data using off the shelf tools
    • Enable legacy applications to access data via NFS
      • Analytic apps can access data without modification
    • Enable object base access to data
      • Modern applications can use object based access to data
  • 110. GlusterFS Deployment - CLI
  • 111. The Gluster Community
    • 300,000+ downloads
      • ~25,000 /month
    • 900+ ‘registered’ deployments
      • 45 countries
    • 2,000+ registered users
      • Mailing lists, Forums, etc.
    • Member: OpenStack, Linux Foundation
    Global adoption
  • 112. The Gluster Community
    • Sounds great. Why are we changing?
      • Only 1 non-Red Hat core contributor
        • There were 2, but he acquired us
      • It's a jungle out there
      • 113. Want to be the software standard for storage
      • 114. Want to be more inclusive, more community-driven
      • 115. Goal: create global ecosystem that includes 3 rd party developers, service providers and more
  • 116. The Gluster Community
    • What is changing?
      • HekaFS / CloudFS being folded into Gluster project
        • HekaFS == GlusterFS + multi-tenancy and SSL for auth and data encryption
        • 117. HekaFS.org
      • Advisory board
        • 9 people who will govern project direction
        • 118. Represents industry and community leaders from Facebook, Citrix, Fedora, and OpenStack
        • 119. Announcement next week (Jan 23)
  • 120. The Gluster Community
    • What is changing?
      • Gluster.org becomes a service
        • Developer section w/ API docs
        • 121. SDK downloads
        • 122. Hosted ancillary projects
      • Published roadmaps
        • Created in public view, with input from community
        • 123. Transparent feature development
  • 124. The Gluster Community
    • Where's the code?
      • Gluster Management Console
        • Java-based GUI
        • 125. First alpha due next week (12/13)
      • GlusterFS 3.3
        • Simultaneous file + object
        • 126. HDFS compatibility
        • 127. Improved self-healing + VM hosting
          • Granular locking
        • Beta 3 due 12/20
  • 128. Your turn - ask our expert Questions and Answers & Resources
    • Additional resources here: http://www.gluster.org/
    • 129. Download GlusterFS: http://www.gluster.org/ download
    • 130. Follow on twitter: @RedHatStorage
    • 131. Like us on Fbook: facebook.com/GlusterInc
    • 132. Read our blog: http://www.gluster.com/blog/
    Contact me: [email_address] Resources