Exploring azure cloud storage
Upcoming SlideShare
Loading in...5
×
 

Exploring azure cloud storage

on

  • 1,278 views

 

Statistics

Views

Total Views
1,278
Views on SlideShare
1,278
Embed Views
0

Actions

Likes
0
Downloads
8
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Why we need?Illustrate with traditional way, file server approach
  • C:\\WindowsAzurePlatformKit\\Demos\\WindowsAzureGuestBookUsingBlobs

Exploring azure cloud storage Exploring azure cloud storage Presentation Transcript

  • K.MohamedFaizal
    Lead Consultant @ NCS (P) Ltd.
    kmdfaizal@gmail.com
    http://faizal-comeacross.blogspot.com
    AzureUG.SG 26th Jan 2011.
    Exploring Windows Azure Cloud Storage
  • About Me
    14
  • Agenda
    Cloud Storage Account Overview
    Tables
    Blobs
    Queues
    Storage Viewer / Explorer Tools
  • Cloud Storage
    Blobs
    Queues
    Tables
    The types of Azure Storage
    • Blobs – A file system
    • Tables – Hierarchical Data Tables
    • Queues – Asynchronous Messaging
    • Drives – NTFS file systems (Blob)
    Each item in storage is stored in triplicate.
  • Make a guess!
    What is the maximum size of data, a storage account can hold?
    1 GB
    100 GB
    1 TB
    100 TB
    Answer: d) 100 TB
    http://blogs.msdn.com/b/windowsazurestorage/archive/2010/05/10/windows-azure-storage-abstractions-and-their-scalability-targets.aspx
  • Windows Azure and SQL Azure
  • Creating a Storage Account
  • With client library
    Automatically referenced in a cloud project
    Supplies a nice .NET API on top of the REST API
    You will need the following:
    Account Name - movieconversion
    Account Shared Key – NOcqFlqUwJPLlm…
    Endpoint - blob.core.windows.net
    -or-
    UseDevelopmentStorage=true
  • Table Storage
  • Windows Azure Tables
    Provides Structured Storage
    Massively Scalable Tables
    Billions of entities (rows) and TBs of data
    Can use thousands of servers as traffic grows
    Highly Available & Durable
    Data is replicated several times
    Familiar and Easy to use API
    ADO.NET Data Services – .NET 3.5 SP1
    .NET classes and LINQ
    REST – with any platform or language
    10
  • Tables
    Is not relational
    Can Not-
    Create foreign key relationships between tables.
    Perform server side joins between tables.
    Create custom indexes on the tables.
    No server side Count(), for example.
    ADO.NET Data Service
    Creates and consumes data via REST-based API's
    The Table Service REST API implements the WCF Data Services API.
  • Table Structure
    Account
    Table
    Account: MovieData
    Table Name: Movies
    Table Name: Customers
    Entity
    Star Wars
    Star Trek
    Fan Boys
    Brian H. Prince
    Jason Argonaut
    Bill Gates
    • Tables store entities.
  • Table Storage Concepts
    Entities
    Tables
    Accounts
    Email =…
    Name = …
    Users
    Email =…
    Name = …
    moviesonline
    Genre =…
    Title = …
    Movies
    Genre =…
    Title = …
    13
  • Table Data Model
    Table
    A storage account can create many tables
    Table name is scoped by account
    Set of entities (i.e. rows)
    Entity
    Set of properties (columns)
    Required properties
    PartitionKey, RowKey and Timestamp
    14
  • Required Entity Properties
    PartitionKey & RowKey
    Uniquely identifies an entity
    Defines the sort order
    Use them to scale your application
    Timestamp
    Read only
    Optimistic Concurrency
    15
  • PartitionKey And Partitions
    PartitionKey
    Used to group entities in the table into partitions
    A table partition
    All entities with same partition key value
    Unit of scale
    Control entity locality
    Row key provides uniqueness within a partition
    16
  • Required Properties
    All entities must have the following properties:
    Timestamp
    PartitionKey
    RowKey
    PartitionKey + RowKey = “primary key”
  • Partitions and Partition Ranges
    Server A
    Table = Movies
    [Action - Comedy)
    Server A
    Table = Movies
    Server B
    Table = Movies
    [Comedy- Western)
    18
  • Table Operations
    Table
    Create
    Query
    Delete
    Entities
    Insert
    Update
    Merge – Partial Update
    Replace – Update entire entity
    Delete
    Query
    Entity Group Transaction (new)
  • Creating a Table
    Tables can be created and destroyed in code.
    Wire up your storage in the on_start method.
    Use a CloudTableClient to access the table.
  • Entity: Inherit the Required Fields
    By inheriting from TableServiceEntity we can easily store our entity in an Azure Table.
  • Create a Context Class
    WCF Data Services needs a context class.
    This represents your connection to the table.
  • Adding an Entity
    Other operations are just like normal ADO.NET Data Services,
    such as delete, update, etc.
  • Deleting an Entity
  • Key Selection: Things to Consider
    Scalability
    Distribute load as much as possible
    PartitionKey is critical for scalability
    Hot partitions can be load balanced
    Query Efficiency & Speed
    Avoid frequent large scans
    Parallelize queries
    25
  • Movie Listing – Solution 1
    Why do I need multiple PartitionKeys?
    Account name as Partition Key
    Movie title as RowKey since movie names need to be sorted
    Category as a separate property
    Does this scale?
    26
  • Movie Listing – Solution 1
    Single partition - Entire table served by one server
    All requests served by that single server
    Does not scale
    Client
    Client
    Request
    Request
    Request
    Request
    Server A
    27
  • Movie Listing – Solution 2
    All movies partitioned by category
    Allows system to load balance hot partitions
    Load distributed
    Better than single partition
    Server A
    Client
    Client
    Request
    Request
    Request
    Request
    Request
    Request
    Request
    Request
    Server B
    28
  • Key Selection: Case Study 2
    Log every transaction into a table for diagnostics
    Scale Write Intensive Scenario
    Logs can be retrieved for a given time range
    29
  • Logging - Solution 1
    Timestamp as Partition Key
    Looks like an obvious choice
    It is not a single partition as time moves forward
    Append only
    Requests to single partition range
    Load balancingdoesnot help
    Server may throttle
    Server A
    Applications
    Client
    Server B
    Request
    Request
    Request
    Request
    30
  • Logging Solution 2 - Distribute "Append Only”
    Prefix timestamp such that load is distributed
    Id of the node logging
    Hash into N buckets
    Write load is now distributed
    Better throughput
    To query logs in time range
    Parallelize it across prefix values
    Server A
    Applications
    Client
    Server B
    Request
    Request
    Request
    Request
    31
  • Table Storage Demo
    demo
  • Overview Blob Storage
  • What is a BLOB
    Binary Large OBject
  • BLOBs in Azure & Size
    Each BLOB can be up to 1TB in size
    You can have as many containers and BLOBs as you want.
    Containers can be created/destroyed on the fly
    http://<account>.blob.core.windows.net/<container>/<blobname>
    http://cohowinery.blob.core.windows.net/images/pic01.jpg
    Account: MovieConversion
    blobs
    Container: Originals
    Container: Completed
    • Job1.mpg
    • Job2.mpg
    • Header.png
    • Job 9.mpg
    • Index.docx
    • Job8.pdf
  • Blob Storage Concepts
    Blob
    Container
    Account
    PIC01.JPG
    images
    MOV1.AVI
    PIC02.JPG
    sally
    movies
  • Windows Azure Storage Services
    New Features
    Blobs – Provide a simple interface for storing named files along with metadata for the file
    New Type of Blob
    Page Blob
    New Blob Operations
    Update Blob
    Copy Blob
    Snapshot Blob
    Lease Blob
    New Ways of Accessing and Serving Blob Content
    Custom Domain Names
    Root Blob Container
    Content Delivery Network access
    Shared Access Signatures
    Drives – Provides durable NTFS volumes for Windows Azure applications to use (Feb 2010)
  • Overview Queue Storage
  • What is a queue?
    Queues are used to store messages
    They are FIFO, and one way
    A queue name must be lower case name, and URL friendly
    Message
    Queue
    Account
    customer ID order ID http://…
    order processing
    cohowinery
    customer ID order ID http://…
  • Account, Queues and Messages
    An account can create many queues
    Queue Name is scoped by the account
    A Queue contains messages
    No limit on number of messages stored in a queue
    Set a limit for message expiration
    Messages
    Message size <= 8 KB
    To store larger data, store data in blob/entity storage, and the blob/entity name in the message
    Message now has dequeue count
    40
  • What Next
    Our next UG Meeting on 30th March 2011
    Microsoft Office Level 21, Singapore
  • Storage Viewer / Explorer Tools
    Azure Storage Explorer
    http://azurestorageexplorer.codeplex.com/
    Myazurestorage.com
    https://myazurestorage.com/
    Cloud Xplorer
    http://clumsyleaf.com/products/cloudxplorer
    Cerebrata Cloud Storage Studio
    http://www.cerebrata.com/products/cloudstoragestudio/