Software Developer Conference 2012 - Paper Presentation - Cloud File Systems

  • 419 views
Uploaded on

Seamless extension of NAS to Cloud Storage using Cloud File System

Seamless extension of NAS to Cloud Storage using Cloud File System

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
419
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
8
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Cloud File System Gateway & Cloud Data Management Interface (CDMI)Author: Imran Khan, Solutions Architect, Calsoft Inc.Presenter: Parag Kulkarni, VP Engineering, Calsoft Inc.2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved.
  • 2. Agenda  Cloud Storage Industry Challenges  Brief about CDMI  Cloud File System(FS)  Cloud FS Architecture  Cloud FS Modules  Cloud FS Solution  Conclusion  Q&A Session 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved. 2
  • 3. Abstract Seamless extension of NAS to Cloud Storage using Cloud File SystemToday NAS stores data on local disks and/or SAN disks. Most enterprises have sufficient filestorage capacity to run their day-to-day operations provided some older data is moved tosecondary storage. But since most data resides on primary storage (or secondary storagewithin enterprise boundaries) it becomes necessary to extend storage capacity for NAS.With Cloud storage becoming more secure, accessible, easy to use and cost effective, it canbe a considered as secondary storage for enterprise NAS – Hierarchical StorageManagement. We can even use cloud storage as primary storage by using enterprise storagedevices for caching to improve cloud data access throughput.Adding CDMI based interfaces to cloud file system enables us to integrate with any cloudstorage provider and store file based data to cloud storage easily. Cloud File Systempresented and implemented by Calsoft integrates with many cloud storage providers usingCDMI. This helps enterprises store file based data to cloud storage and provides throughputsimilar to local NAS by using efficient caching techniques. 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved. 3
  • 4. Learning Objectives Challenges  To store ever growing data and optimally manage storage capacity  Hierarchal Storage Management across enterprise storage and cloud storage  Measure/monitor user guarantees and SLAs and map them over multiple clouds  Optimizing storage capacity between on-premise and cloud storage pools  To migrate between cloud storage platforms Solution Adoption of Cloud Storage CDMI – move to build an open standards for storing data in the Cloud No impact on existing users/apps using NAS Abstract policy engine to monitor and map the SLAs 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved. 4
  • 5. Cloud Storage Industry Challenges Access Bandwidth, Delay And Disruption Of Service Common Interface To Multiple Clouds Data Security Data Transfer Policy Auxiliary Features 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved. 5
  • 6. Cloud Data Management Interface CDMI  A protocol for self-provisioning, administering and accessing cloud storage.  It defines RESTful HTTP operations for assessing capabilities of cloud storage system and exporting data via other protocols such as CIFS and NFS CDMI Benefits  To manage containers, domains, security access  Easy of monitoring / billing  For storage that is functionally accessible by legacy or proprietary protocols 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved. 6
  • 7. Calsoft’s Cloud File System Leveraging CDMI to provide a common interface to interact with multiple clouds Cloud Access Cloud Request Multiple Cloud Framework • File system • Convert the • Enables interface to the filesystem interaction with cloud access. demand into multiple clouds, cloud requests while abstracting • Filesystem cache out many the data from the • Convert the data operations cloud to provide a objects back to quicker access common file • Dynamically model changes support • Filesystem for various cloud interface vendors provided to the clients using NFS, • Provides a set of CIFS etc policies to control access patterns 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved. 7
  • 8. Cloud File System Architecture User 1 C CDMI I CDMI Compliant User 2 F CLOUD DATA 3rd party Cloud Cloud 1 Cloud ACCESSS Storage Plug-in Storage S Cloud 2 / LAYER SOAP / REST User 3 N / WebDAV F USER MANAGEMENT Cloud 1 - CDMI S POLICY MANAGEMENT Non Compliant Cloud Storage User 4 Etc. S E …… R LVM, Disk Driver, RAID, etc. V Cloud 2 - CDMI User n E Non Compliant R Cloud Storage CLOUD FILE SYSTEM Local / SAN Disks 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved. 8
  • 9. Cloud File System Modules Cloud S3 PluginsCloud interface Otherand the policy CFS User Space Cloudengine Interface Other CFS user space NIFS/ & Policy and Local FS CIFS Command translation Local FS wrapper Engine Other wrapper User User Cloud kernel NIFS/ FS Command CIFS ( CFS) Layer 1 Functionality Translation Kernnel Layer 2 Functionality Local Cache FS (LCFS) Cloud FS – from cache or not? Ext 3 reiser other NFS / CIFS 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved. 9
  • 10. User Space Vs. Kernel Space• Most of the file systems in user space (using FUSE) are designed so for the ease of developing and maintaining them. Example - s3fs for Amazon S3 cloud.• Doesn’t mean that performance is guaranteed.• User space FS makes one or sometimes more than one data copies.• When using a cache, in case of a hit, FUSE will still need a context switch and data copy.• Linus Trovalds – ‘I think that arguing that something _can_ be done with fuse, and thus _should_ be done with fuse is just ridiculous.’• Polpulating the local cache can be done by just a command, why pass a buffer like fuse does.• Layered functionality inside the FS (for ex: splitting) is easy to implement and could prove useful. 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved. 10
  • 11. Policy Engine • Pricing models that most clouds use – • Storage based - $/GB • Request based - $/1000 requests. • Data transfer based - $/GB • Other QoS parameters that determine choice of cloud • Easy provisioning • Multi-tenancy • Security • Reliability • These parameters especially pricing is tracked by the service provider. There is no easy way for user to track these parameters. • Also, there is no standard or specification that defines these parameters. • The policy engine module, proves to be an efficient solution • To try and define these parameters across multiple clouds • To monitor, keep track of these parameters • Allows a rule based framework to control the access to these clouds based on the QoS they provide. • In future maybe these QoS parameters can be standardized • And made accessible via APIs, enabling users to program against these parameters 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved. 11
  • 12. Cloud File System Solution Industry Challenges Calsoft Solution The policy engine in cloud interface module can be used to Access bandwidth, delay distribute or replicate data across multiple clouds. Loss of service and disruption of service from one cloud will not hamper access to any data. The plugins to interface with different clouds supporting different Common interface to communication protocols can be written independently and multiple cloud loaded at run time The policy engine can also select different security algorithms based on different clouds, which can be applied to the data while Security sending out over the wire. It is more efficient since it is out of band for a cache hit scenario The Policy engine is user controlled and xml based. The rules can Data transfer policy be as simple and as comprehensive as needed The cloud interface and plugins can do book keeping that can be Auxiliary features used to verify amount of data transferred and compare the cost of that data transfer against the billed amount 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved. 12
  • 13. Conclusion Cloud File System is an idea that has taken into consideration current events in Cloud world related to Data storage as a Service (DaaS) It is a prediction of how infrastructure around cloud services and management has changed. This model that will improve performance, will enable seamless transitions across CDMI compliant and non-compliant clouds for large enterprises with very less hassle. 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved. 13
  • 14. Presenter Biography Parag Kulkarni – VP Engineering, Calsoft Inc.  A veteran of storage industry  More than 19 years of experience in architecting and developing products  Key strength lies in quickly understanding product requirements and translating them into architectural and engineering specs for implementation.  Led the engineering team at Calsoft.  Led the development of Database Editions product at Veritas (Symantec)  A key contributing member at leading storage companies like Informix (IBM).  Masters of Technology in Computer Science from IIT Roorkee  Degree in Industrial Management from University of Indore, India. 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved.
  • 15. Author Biography Imran Khan – Solutions Architect, Calsoft Inc.  A veteran of storage industry  More than 8 years of experience in architecting and developing products  Has dealt with products ranging from backup and replication, SAN simulators, multipathing, SMI-S, filesystems, journaling, link aggregation protocols.  Key strength is the ability to have holistic view across stacks of different functionality and their interaction.  Bachelors in Computers Engineering from University of Pune, India. 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved.
  • 16. Thank You Questions & Answers Contact info Parag Kulkarni VP Engineering, Calsoft Inc. Email: parag.kulkarni@calsoftinc.com Phone: +1 (408) 834 7086 2012 Storage Developer Conference. © Calsoft Inc.. All Rights Reserved. 16