Exploring Windows Azure Cloud Storage


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Why we need?Illustrate with traditional way, file server approach
  • C:\\WindowsAzurePlatformKit\\Demos\\WindowsAzureGuestBookUsingBlobs
  • Exploring Windows Azure Cloud Storage

    1. 1. K.MohamedFaizal<br />Lead Consultant @ NCS (P) Ltd.<br />kmdfaizal@gmail.com<br />http://faizal-comeacross.blogspot.com<br />AzureUG.SG 26th Jan 2011.<br />Exploring Windows Azure Cloud Storage<br />
    2. 2. About Me<br />14<br />
    3. 3. Agenda<br />Cloud Storage Account Overview<br />Tables<br />Blobs<br />Queues<br />Storage Viewer / Explorer Tools<br />
    4. 4. Cloud Storage<br />Blobs<br />Queues<br />Tables<br />The types of Azure Storage<br /><ul><li>Blobs – A file system
    5. 5. Tables – Hierarchical Data Tables
    6. 6. Queues – Asynchronous Messaging
    7. 7. Drives – NTFS file systems (Blob)</li></ul>Each item in storage is stored in triplicate.<br />
    8. 8. Make a guess!<br />What is the maximum size of data, a storage account can hold?<br />1 GB<br />100 GB<br />1 TB<br />100 TB<br />Answer: d) 100 TB<br />http://blogs.msdn.com/b/windowsazurestorage/archive/2010/05/10/windows-azure-storage-abstractions-and-their-scalability-targets.aspx<br />
    9. 9. Windows Azure and SQL Azure <br />
    10. 10. Creating a Storage Account<br />
    11. 11. With client library<br />Automatically referenced in a cloud project<br />Supplies a nice .NET API on top of the REST API<br />You will need the following:<br />Account Name - movieconversion<br />Account Shared Key – NOcqFlqUwJPLlm…<br />Endpoint - blob.core.windows.net<br />-or-<br />UseDevelopmentStorage=true<br />
    12. 12. Table Storage<br />
    13. 13. Windows Azure Tables<br />Provides Structured Storage<br />Massively Scalable Tables<br />Billions of entities (rows) and TBs of data<br />Can use thousands of servers as traffic grows<br />Highly Available & Durable<br />Data is replicated several times<br />Familiar and Easy to use API<br />ADO.NET Data Services – .NET 3.5 SP1<br />.NET classes and LINQ<br />REST – with any platform or language<br />10<br />
    14. 14. Tables<br />Is not relational<br />Can Not-<br />Create foreign key relationships between tables.<br />Perform server side joins between tables.<br />Create custom indexes on the tables.<br />No server side Count(), for example.<br />ADO.NET Data Service<br />Creates and consumes data via REST-based API's<br />The Table Service REST API implements the WCF Data Services API.<br />
    15. 15. Table Structure<br />Account<br />Table<br />Account: MovieData<br />Table Name: Movies<br />Table Name: Customers<br />Entity<br />Star Wars<br />Star Trek<br />Fan Boys<br />Brian H. Prince<br />Jason Argonaut<br />Bill Gates<br /><ul><li>Tables store entities.</li></li></ul><li>Table Storage Concepts<br />Entities<br />Tables<br />Accounts<br />Email =…<br />Name = …<br />Users<br />Email =…<br />Name = …<br />moviesonline<br />Genre =…<br />Title = …<br />Movies<br />Genre =…<br />Title = …<br />13<br />
    16. 16. Table Data Model<br />Table<br />A storage account can create many tables<br />Table name is scoped by account<br />Set of entities (i.e. rows)<br />Entity<br />Set of properties (columns)<br />Required properties<br />PartitionKey, RowKey and Timestamp<br />14<br />
    17. 17. Required Entity Properties<br />PartitionKey & RowKey<br />Uniquely identifies an entity<br />Defines the sort order<br />Use them to scale your application<br />Timestamp <br />Read only<br />Optimistic Concurrency<br />15<br />
    18. 18. PartitionKey And Partitions<br />PartitionKey<br />Used to group entities in the table into partitions<br />A table partition <br />All entities with same partition key value<br />Unit of scale<br />Control entity locality<br />Row key provides uniqueness within a partition<br />16<br />
    19. 19. Required Properties<br />All entities must have the following properties:<br />Timestamp<br />PartitionKey<br />RowKey<br />PartitionKey + RowKey = “primary key”<br />
    20. 20. Partitions and Partition Ranges<br />Server A<br />Table = Movies<br />[Action - Comedy)<br />Server A<br />Table = Movies<br />Server B<br />Table = Movies<br />[Comedy- Western)<br />18<br />
    21. 21. Table Operations<br />Table<br />Create<br />Query<br />Delete<br />Entities<br />Insert<br />Update <br />Merge – Partial Update<br />Replace – Update entire entity<br />Delete<br />Query<br />Entity Group Transaction (new)<br />
    22. 22. Creating a Table<br />Tables can be created and destroyed in code.<br />Wire up your storage in the on_start method.<br />Use a CloudTableClient to access the table.<br />
    23. 23. Entity: Inherit the Required Fields<br />By inheriting from TableServiceEntity we can easily store our entity in an Azure Table.<br />
    24. 24. Create a Context Class<br />WCF Data Services needs a context class. <br />This represents your connection to the table.<br />
    25. 25. Adding an Entity<br />Other operations are just like normal ADO.NET Data Services,<br />such as delete, update, etc.<br />
    26. 26. Deleting an Entity<br />
    27. 27. Key Selection: Things to Consider<br />Scalability<br />Distribute load as much as possible<br />PartitionKey is critical for scalability<br />Hot partitions can be load balanced<br />Query Efficiency & Speed<br />Avoid frequent large scans<br />Parallelize queries<br />25<br />
    28. 28. Movie Listing – Solution 1<br />Why do I need multiple PartitionKeys?<br />Account name as Partition Key <br />Movie title as RowKey since movie names need to be sorted<br />Category as a separate property<br />Does this scale?<br />26<br />
    29. 29. Movie Listing – Solution 1<br />Single partition - Entire table served by one server<br />All requests served by that single server<br />Does not scale<br />Client<br />Client<br />Request<br />Request<br />Request<br />Request<br />Server A<br />27<br />
    30. 30. Movie Listing – Solution 2<br />All movies partitioned by category<br />Allows system to load balance hot partitions<br />Load distributed<br />Better than single partition<br />Server A<br />Client<br />Client<br />Request<br />Request<br />Request<br />Request<br />Request<br />Request<br />Request<br />Request<br />Server B<br />28<br />
    31. 31. Key Selection: Case Study 2<br />Log every transaction into a table for diagnostics<br />Scale Write Intensive Scenario<br />Logs can be retrieved for a given time range<br />29<br />
    32. 32. Logging - Solution 1<br />Timestamp as Partition Key <br />Looks like an obvious choice<br />It is not a single partition as time moves forward <br />Append only<br />Requests to single partition range<br />Load balancingdoesnot help<br />Server may throttle<br />Server A<br />Applications<br />Client<br />Server B<br />Request<br />Request<br />Request<br />Request<br />30<br />
    33. 33. Logging Solution 2 - Distribute "Append Only”<br />Prefix timestamp such that load is distributed<br />Id of the node logging<br />Hash into N buckets<br />Write load is now distributed <br />Better throughput<br />To query logs in time range<br />Parallelize it across prefix values<br />Server A<br />Applications<br />Client<br />Server B<br />Request<br />Request<br />Request<br />Request<br />31<br />
    34. 34. Table Storage Demo<br />demo<br />
    35. 35. Overview Blob Storage<br />
    36. 36. What is a BLOB<br />Binary Large OBject<br />
    37. 37. BLOBs in Azure & Size<br />Each BLOB can be up to 1TB in size<br />You can have as many containers and BLOBs as you want.<br />Containers can be created/destroyed on the fly<br />http://<account>.blob.core.windows.net/<container>/<blobname><br />http://cohowinery.blob.core.windows.net/images/pic01.jpg<br />Account: MovieConversion<br />blobs<br />Container: Originals<br />Container: Completed<br /><ul><li>Job1.mpg
    38. 38. Job2.mpg
    39. 39. Header.png
    40. 40. Job 9.mpg
    41. 41. Index.docx
    42. 42. Job8.pdf</li></li></ul><li>Blob Storage Concepts<br />Blob<br />Container<br />Account<br />PIC01.JPG<br />images<br />MOV1.AVI<br />PIC02.JPG<br />sally<br />movies<br />
    43. 43. Windows Azure Storage Services<br />New Features<br />Blobs – Provide a simple interface for storing named files along with metadata for the file<br />New Type of Blob<br />Page Blob<br />New Blob Operations<br />Update Blob<br />Copy Blob<br />Snapshot Blob<br />Lease Blob<br />New Ways of Accessing and Serving Blob Content<br />Custom Domain Names<br />Root Blob Container<br />Content Delivery Network access<br />Shared Access Signatures <br />Drives – Provides durable NTFS volumes for Windows Azure applications to use (Feb 2010)<br />
    44. 44. Overview Queue Storage<br />
    45. 45. What is a queue? <br />Queues are used to store messages<br />They are FIFO, and one way<br />A queue name must be lower case name, and URL friendly<br />Message<br />Queue<br />Account<br /> customer ID order ID http://…<br />order processing<br />cohowinery<br /> customer ID order ID http://…<br />
    46. 46. Account, Queues and Messages<br />An account can create many queues<br />Queue Name is scoped by the account<br />A Queue contains messages<br />No limit on number of messages stored in a queue<br />Set a limit for message expiration<br />Messages<br />Message size <= 8 KB<br />To store larger data, store data in blob/entity storage, and the blob/entity name in the message<br />Message now has dequeue count<br />40<br />
    47. 47.
    48. 48. What Next<br />Our next UG Meeting on 30th March 2011<br />Microsoft Office Level 21, Singapore<br />
    49. 49. Storage Viewer / Explorer Tools<br />Azure Storage Explorer<br />http://azurestorageexplorer.codeplex.com/<br />Myazurestorage.com<br />https://myazurestorage.com/<br />Cloud Xplorer<br />http://clumsyleaf.com/products/cloudxplorer<br />Cerebrata Cloud Storage Studio<br />http://www.cerebrata.com/products/cloudstoragestudio/<br />