Cloud Storage Blobs Queues Tables The types of Azure Storage
Blobs – A file system
Tables – Hierarchical Data Tables
Queues – Asynchronous Messaging
Drives – NTFS file systems (Blob)
Each item in storage is stored in triplicate.
Make a guess! What is the maximum size of data, a storage account can hold? 1 GB 100 GB 1 TB 100 TB Answer: d) 100 TB http://blogs.msdn.com/b/windowsazurestorage/archive/2010/05/10/windows-azure-storage-abstractions-and-their-scalability-targets.aspx
Windows Azure and SQL Azure
Creating a Storage Account
With client library Automatically referenced in a cloud project Supplies a nice .NET API on top of the REST API You will need the following: Account Name - movieconversion Account Shared Key – NOcqFlqUwJPLlm… Endpoint - blob.core.windows.net -or- UseDevelopmentStorage=true
Windows Azure Tables Provides Structured Storage Massively Scalable Tables Billions of entities (rows) and TBs of data Can use thousands of servers as traffic grows Highly Available & Durable Data is replicated several times Familiar and Easy to use API ADO.NET Data Services – .NET 3.5 SP1 .NET classes and LINQ REST – with any platform or language 10
Tables Is not relational Can Not- Create foreign key relationships between tables. Perform server side joins between tables. Create custom indexes on the tables. No server side Count(), for example. ADO.NET Data Service Creates and consumes data via REST-based API's The Table Service REST API implements the WCF Data Services API.
Table Structure Account Table Account: MovieData Table Name: Movies Table Name: Customers Entity Star Wars Star Trek Fan Boys Brian H. Prince Jason Argonaut Bill Gates
Tables store entities.
Table Storage Concepts Entities Tables Accounts Email =… Name = … Users Email =… Name = … moviesonline Genre =… Title = … Movies Genre =… Title = … 13
Table Data Model Table A storage account can create many tables Table name is scoped by account Set of entities (i.e. rows) Entity Set of properties (columns) Required properties PartitionKey, RowKey and Timestamp 14
Required Entity Properties PartitionKey & RowKey Uniquely identifies an entity Defines the sort order Use them to scale your application Timestamp Read only Optimistic Concurrency 15
PartitionKey And Partitions PartitionKey Used to group entities in the table into partitions A table partition All entities with same partition key value Unit of scale Control entity locality Row key provides uniqueness within a partition 16
Required Properties All entities must have the following properties: Timestamp PartitionKey RowKey PartitionKey + RowKey = “primary key”
Partitions and Partition Ranges Server A Table = Movies [Action - Comedy) Server A Table = Movies Server B Table = Movies [Comedy- Western) 18
Creating a Table Tables can be created and destroyed in code. Wire up your storage in the on_start method. Use a CloudTableClient to access the table.
Entity: Inherit the Required Fields By inheriting from TableServiceEntity we can easily store our entity in an Azure Table.
Create a Context Class WCF Data Services needs a context class. This represents your connection to the table.
Adding an Entity Other operations are just like normal ADO.NET Data Services, such as delete, update, etc.
Deleting an Entity
Key Selection: Things to Consider Scalability Distribute load as much as possible PartitionKey is critical for scalability Hot partitions can be load balanced Query Efficiency & Speed Avoid frequent large scans Parallelize queries 25
Movie Listing – Solution 1 Why do I need multiple PartitionKeys? Account name as Partition Key Movie title as RowKey since movie names need to be sorted Category as a separate property Does this scale? 26
Movie Listing – Solution 1 Single partition - Entire table served by one server All requests served by that single server Does not scale Client Client Request Request Request Request Server A 27
Movie Listing – Solution 2 All movies partitioned by category Allows system to load balance hot partitions Load distributed Better than single partition Server A Client Client Request Request Request Request Request Request Request Request Server B 28
Key Selection: Case Study 2 Log every transaction into a table for diagnostics Scale Write Intensive Scenario Logs can be retrieved for a given time range 29
Logging - Solution 1 Timestamp as Partition Key Looks like an obvious choice It is not a single partition as time moves forward Append only Requests to single partition range Load balancingdoesnot help Server may throttle Server A Applications Client Server B Request Request Request Request 30
Logging Solution 2 - Distribute "Append Only” Prefix timestamp such that load is distributed Id of the node logging Hash into N buckets Write load is now distributed Better throughput To query logs in time range Parallelize it across prefix values Server A Applications Client Server B Request Request Request Request 31
Table Storage Demo demo
Overview Blob Storage
What is a BLOB Binary Large OBject
BLOBs in Azure & Size Each BLOB can be up to 1TB in size You can have as many containers and BLOBs as you want. Containers can be created/destroyed on the fly http://<account>.blob.core.windows.net/<container>/<blobname> http://cohowinery.blob.core.windows.net/images/pic01.jpg Account: MovieConversion blobs Container: Originals Container: Completed
Windows Azure Storage Services New Features Blobs – Provide a simple interface for storing named files along with metadata for the file New Type of Blob Page Blob New Blob Operations Update Blob Copy Blob Snapshot Blob Lease Blob New Ways of Accessing and Serving Blob Content Custom Domain Names Root Blob Container Content Delivery Network access Shared Access Signatures Drives – Provides durable NTFS volumes for Windows Azure applications to use (Feb 2010)
Overview Queue Storage
What is a queue? Queues are used to store messages They are FIFO, and one way A queue name must be lower case name, and URL friendly Message Queue Account customer ID order ID http://… order processing cohowinery customer ID order ID http://…
Account, Queues and Messages An account can create many queues Queue Name is scoped by the account A Queue contains messages No limit on number of messages stored in a queue Set a limit for message expiration Messages Message size <= 8 KB To store larger data, store data in blob/entity storage, and the blob/entity name in the message Message now has dequeue count 40
What Next Our next UG Meeting on 30th March 2011 Microsoft Office Level 21, Singapore
Storage Viewer / Explorer Tools Azure Storage Explorer http://azurestorageexplorer.codeplex.com/ Myazurestorage.com https://myazurestorage.com/ Cloud Xplorer http://clumsyleaf.com/products/cloudxplorer Cerebrata Cloud Storage Studio http://www.cerebrata.com/products/cloudstoragestudio/