1) Azure storage includes three main types: blobs for unstructured data like files, tables for structured non-relational data, and queues for asynchronous messaging. 2) Blobs can be up to 1TB in size and are stored in containers within a storage account. Tables require entities to have a partition key and row key for indexing. Queues allow storing messages for asynchronous processing. 3) Keys are important for scalability in tables - the partition key controls how loads are distributed. Entities with the same partition key are stored together.

1. K.MohamedFaizalLead Consultant @ NCS (P) Ltd.kmdfaizal@gmail.comhttp://faizal-comeacross.blogspot.comAzureUG.SG 26th Jan 2011.Exploring Windows Azure Cloud Storage

2. About Me14

3. AgendaCloud Storage Account OverviewTablesBlobsQueuesStorage Viewer / Explorer Tools

4. Cloud StorageBlobsQueuesTablesThe types of Azure StorageBlobs – A file system

5. Tables – Hierarchical Data Tables

6. Queues – Asynchronous Messaging

7. Drives – NTFS file systems (Blob)Each item in storage is stored in triplicate.

8. Make a guess!What is the maximum size of data, a storage account can hold?1 GB100 GB1 TB100 TBAnswer: d) 100 TBhttp://blogs.msdn.com/b/windowsazurestorage/archive/2010/05/10/windows-azure-storage-abstractions-and-their-scalability-targets.aspx

9. Windows Azure and SQL Azure

10. Creating a Storage Account

11. With client libraryAutomatically referenced in a cloud projectSupplies a nice .NET API on top of the REST APIYou will need the following:Account Name - movieconversionAccount Shared Key – NOcqFlqUwJPLlm…Endpoint - blob.core.windows.net-or-UseDevelopmentStorage=true

12. Table Storage

13. Windows Azure TablesProvides Structured StorageMassively Scalable TablesBillions of entities (rows) and TBs of dataCan use thousands of servers as traffic growsHighly Available & DurableData is replicated several timesFamiliar and Easy to use APIADO.NET Data Services – .NET 3.5 SP1.NET classes and LINQREST – with any platform or language10

14. TablesIs not relationalCan Not-Create foreign key relationships between tables.Perform server side joins between tables.Create custom indexes on the tables.No server side Count(), for example.ADO.NET Data ServiceCreates and consumes data via REST-based API'sThe Table Service REST API implements the WCF Data Services API.

15. Table StructureAccountTableAccount: MovieDataTable Name: MoviesTable Name: CustomersEntityStar WarsStar TrekFan BoysBrian H. PrinceJason ArgonautBill GatesTables store entities.Table Storage ConceptsEntitiesTablesAccountsEmail =…Name = …UsersEmail =…Name = …moviesonlineGenre =…Title = …MoviesGenre =…Title = …13

16. Table Data ModelTableA storage account can create many tablesTable name is scoped by accountSet of entities (i.e. rows)EntitySet of properties (columns)Required propertiesPartitionKey, RowKey and Timestamp14

17. Required Entity PropertiesPartitionKey & RowKeyUniquely identifies an entityDefines the sort orderUse them to scale your applicationTimestamp Read onlyOptimistic Concurrency15

18. PartitionKey And PartitionsPartitionKeyUsed to group entities in the table into partitionsA table partition All entities with same partition key valueUnit of scaleControl entity localityRow key provides uniqueness within a partition16

19. Required PropertiesAll entities must have the following properties:TimestampPartitionKeyRowKeyPartitionKey + RowKey = “primary key”

20. Partitions and Partition RangesServer ATable = Movies[Action - Comedy)Server ATable = MoviesServer BTable = Movies[Comedy- Western)18

21. Table OperationsTableCreateQueryDeleteEntitiesInsertUpdate Merge – Partial UpdateReplace – Update entire entityDeleteQueryEntity Group Transaction (new)

22. Creating a TableTables can be created and destroyed in code.Wire up your storage in the on_start method.Use a CloudTableClient to access the table.

23. Entity: Inherit the Required FieldsBy inheriting from TableServiceEntity we can easily store our entity in an Azure Table.

24. Create a Context ClassWCF Data Services needs a context class. This represents your connection to the table.

25. Adding an EntityOther operations are just like normal ADO.NET Data Services,such as delete, update, etc.

26. Deleting an Entity

27. Key Selection: Things to ConsiderScalabilityDistribute load as much as possiblePartitionKey is critical for scalabilityHot partitions can be load balancedQuery Efficiency & SpeedAvoid frequent large scansParallelize queries25

28. Movie Listing – Solution 1Why do I need multiple PartitionKeys?Account name as Partition Key Movie title as RowKey since movie names need to be sortedCategory as a separate propertyDoes this scale?26

29. Movie Listing – Solution 1Single partition - Entire table served by one serverAll requests served by that single serverDoes not scaleClientClientRequestRequestRequestRequestServer A27

30. Movie Listing – Solution 2All movies partitioned by categoryAllows system to load balance hot partitionsLoad distributedBetter than single partitionServer AClientClientRequestRequestRequestRequestRequestRequestRequestRequestServer B28

31. Key Selection: Case Study 2Log every transaction into a table for diagnosticsScale Write Intensive ScenarioLogs can be retrieved for a given time range29

32. Logging - Solution 1Timestamp as Partition Key Looks like an obvious choiceIt is not a single partition as time moves forward Append onlyRequests to single partition rangeLoad balancingdoesnot helpServer may throttleServer AApplicationsClientServer BRequestRequestRequestRequest30

33. Logging Solution 2 - Distribute "Append Only”Prefix timestamp such that load is distributedId of the node loggingHash into N bucketsWrite load is now distributed Better throughputTo query logs in time rangeParallelize it across prefix valuesServer AApplicationsClientServer BRequestRequestRequestRequest31

34. Table Storage Demodemo

35. Overview Blob Storage

36. What is a BLOBBinary Large OBject

37. BLOBs in Azure & SizeEach BLOB can be up to 1TB in sizeYou can have as many containers and BLOBs as you want.Containers can be created/destroyed on the flyhttp://<account>.blob.core.windows.net/<container>/<blobname>http://cohowinery.blob.core.windows.net/images/pic01.jpgAccount: MovieConversionblobsContainer: OriginalsContainer: CompletedJob1.mpg

38. Job2.mpg

39. Header.png

40. Job 9.mpg

41. Index.docx

42. Job8.pdfBlob Storage ConceptsBlobContainerAccountPIC01.JPGimagesMOV1.AVIPIC02.JPGsallymovies

43. Windows Azure Storage ServicesNew FeaturesBlobs – Provide a simple interface for storing named files along with metadata for the fileNew Type of BlobPage BlobNew Blob OperationsUpdate BlobCopy BlobSnapshot BlobLease BlobNew Ways of Accessing and Serving Blob ContentCustom Domain NamesRoot Blob ContainerContent Delivery Network accessShared Access Signatures Drives – Provides durable NTFS volumes for Windows Azure applications to use (Feb 2010)

44. Overview Queue Storage

45. What is a queue? Queues are used to store messagesThey are FIFO, and one wayA queue name must be lower case name, and URL friendlyMessageQueueAccount customer ID order ID http://…order processingcohowinery customer ID order ID http://…

46. Account, Queues and MessagesAn account can create many queuesQueue Name is scoped by the accountA Queue contains messagesNo limit on number of messages stored in a queueSet a limit for message expirationMessagesMessage size <= 8 KBTo store larger data, store data in blob/entity storage, and the blob/entity name in the messageMessage now has dequeue count40

48. What NextOur next UG Meeting on 30th March 2011Microsoft Office Level 21, Singapore

49. Storage Viewer / Explorer ToolsAzure Storage Explorerhttp://azurestorageexplorer.codeplex.com/Myazurestorage.comhttps://myazurestorage.com/Cloud Xplorerhttp://clumsyleaf.com/products/cloudxplorerCerebrata Cloud Storage Studiohttp://www.cerebrata.com/products/cloudstoragestudio/