Design Considerations For Storing With Windows Azure

Design considerations for storing data in the cloud with Windows Azure
Eric Nelson
Microsoft UK
Blog: http://geekswithblogs.net/iupdateable
Twitter: http://twitter.com/ericnel and http://twitter.com/ukmsdn
Podcast: http://bit.ly/msdnpodcast
Newsletter: http://msdn.microsoft.com/en-gb/flash
Slides, links and background “diary” posts can be found on my blog

Windows Azure Platform 101
Storage in the Cloud
Blobs
Tables
Relational
Queues
Lesson learned
Agenda

Windows Azure PLATFORM 101
Just in case you had something better to do over the last 18months

3 Important Services
3 Critical Concepts
Windows Azure
Compute and Storage
SQL Azure
Storage
.NET Services
Connecting
Computation
Web and Worker
Storage
Table, Blob, Relational
Messaging
Queues, Service Bus

A simple site
“Wow! What a great site!”
Database
Request
Web Tier
B/L Tier
Browser
Response

Under load
Browser
Browser
Database
Web Tier
B/L Tier
Browser
“Server Busy”
Browser
Browser

Under load
Browser
Browser
Database
Web Tier
B/L Tier
Browser
“Timeout”
Browser
Browser

Solve using on-premise
Browser
p1 p2 p3
Web Tier
N L B
B/L Tier
N L B
Browser
Database
Web Tier
Browser
B/L Tier
Browser
Web Tier
B/L Tier
Browser

However…
p1 p2 p3
“Not so great now…”
Web Tier
N L B
B/L Tier
N L B
Database
Web Tier
Browser
B/L Tier
Web Tier
B/L Tier
“That took a lot of work - and money!”
“Hmmm... Most of this stuff is sitting idle...”

Solve using the Cloud aka Windows Azure Platform
Browser
p1 p2 p3
Web Role
N L B
Worker Role
N L B
Browser
AzureStorage
Web Role
Browser
Worker Role
Worker Role
Browser
Web Role
Browser
You don’t see this bit
or…
Maybe you do

Solve using the Cloud aka Windows Azure Platform
SQLAzure
Browser
p1 p2 p3
Web Role
N L B
Worker Role
N L B
Browser
AzureStorage
Web Role
Browser
Worker Role
Worker Role
Browser
Web Role
Browser
Ok, you definitely do

Demo: Windows Azure Portal

Storage in the Cloud…
Windows Azure Storage and SQL Azure

Blobs, Tables, Relational

Blobs stored in Containers
1 or more Containers per account
Scoping is at container level
…/Container/blobpath
Blobs
Capacity 50GB in CTP
Metadata, accessed independently
Private or Public container access
Blobs

Put a Blob
Blob Container
PutBlob
PUT http://account.blob.core.windows./net/containername/blobname
Azure
Blob Storage
REST API
Client
http://account.blob.core.windows.net/containername/blobname

Get a Blob
Blob Container
Azure
Blob Storage
REST API
Client
GetBlob
GET http://account.blob.core.windows./net/containername/blobname

Get part of a Blob
Blob Container
Azure
Blob Storage
REST API
Client
GetBlob
GET http://account.blob.core.windows./net/containername/blobname
Range: bytes=329300 - 730000

Put a LARGE Blob
PutBlock(blobname, blockid1, data)
Blob Container
PutBlock(blobname, blockid7, data)
PutBlockList(blobname, blockid1, …, blockidN)
Azure
Blob Storage
REST API
Client

Provides structured storage
Massively scalable tables (TBs of data)
Self scaling
Highly available
Durable
Familiar and easy-to-use API, layered
.NET classes and LINQ
ADO.NET Data Services – .NET 3.5 SP1
REST – with any platform or language
Introduction to Tables

No join
No group by
No order by
“No Schema”
Not a Relational Database

Table
A Table is a set of Entities (rows)
An Entity is a set of Properties (columns)
Entity
Two “key” properties form unique ID
PartitionKey – enables scale
RowKey – uniquely ID within a partition
Data Model

Key Example – Blog Posts
Partition 1
Partition 2
Getting all of dunnry’s blog posts is fast
Single partition
Getting all posts after 2008-03-27 is slow
Traverse all partitions

Query a Table
REST:
GET http://account.table.core.windows.net/Customer?$filter=%20PartitionKey%20eq%20value
LINQ:
var customers = from o in context.CreateQuery<customer>(“Customer”) where o.PartitionKey == value select o;
Azure
Table Storage
Worker Role
http://account.table.core.windows.net

Tradeoff between locality and scalability
Considerations
Entity group transactions
Query efficiency
Scalability
Flexible Partitioning
Choosing a Partition Key

Pick potential keys (common query filters)
Order keys by importance
If needed, include an additional unique key
Use two most important keys as PK, RK
Consider concatenating to form keys
A Method of Choosing Keys

Non-key queries are scans
Improve performance by scoping
Usually by partition key
But what about by table?
3 tables
Top 1,000 popular items
Top 10,000 popular items
Everything
Now arbitrary “top 1,000” queries are fast
Better locality than clever partition keys
Write many is one approach

Demo: Windows Azure Storage

Lessons LearnedAzure Storage
Azure tables are *not* a relational database
Requires a mind shift
Azure tables scale
3 - 9s availability
Azure tables support exactly one key
PartitionKey + RowKey
Case Matters
No foreign keys
No referential integrity
No stored procedures

Lessons LearnedAzure Storage
Azure Storage Client Library
No longer just a “sample”
Azure storage is available via REST
Not limited to Azure hosted apps
Not limited to Microsoft platform or tools
Getting the signature correct is the hard part

Lessons LearnedAzure Storage - RESTful
REST is *not* TDS
Be prepared to parse
LINQ and XML classes help
Sometimes, string parsing is the best choice
Azure storage names are picky
So are Azure key values
It’s possible to create an entity in a table and not be able to update or delete it

Lessons LearnedAzure Storage – Roundtrips are expensive
Often better to pull back more than you need vs. multiple roundtrips
LINQ on results in memory is fast & flexible
foreach works well too
Sort and cache tables on the web tier

Lessons LearnedAzure Storage – Entity Group Transactions
Different Entity types in the same table
E.g. PK = CustomerId
Customer, Order and OrderDetails in the same table

SQL Azure (July 2009)aka SQL Data Servicesaka SQL Server Data Services

On Premise Programming Model
This is what we do on-premise...
Data
TDS
RDBMS
Client
SQLServer

Same for the cloud?
So, is this is what we would like to do in the cloud...
Data
TDS
RDBMS
Client
SQL Server

SQL Azure can do this
Data
TDS
RDBMS
Client
SQL Azure

SQL Azure can also do this
HTTP
TDS
RDBMS
Browser
Web Role
SQL Azure

And this!
Queue
TDS
HTTP
RDBMS
Browser
Web Role
Worker Role
SQL Azure

Which means you can easily migrate from this
“The Data Center”
TDS
HTTP
RDBMS
Browser
Web Tier
Bus. Logic
SQL Server

To this… Windows Azure and SQL Azure
“The Cloud”
Queue
TDS
HTTP
RDBMS
Browser
Web Role
Worker Role
SQL Azure

Demo: SQL Azure

Lessons LearnedSQL Azure
From the database “down” it’s just SQL Server
Well, almost …
Many tools don’t work today
System catalog is different
Above the database is taken care of for you
You can’t really change anything

Tooling
SSMS partially works – “good enough”
Can not create connection using Visual Studio designer
Other tools may work better
No BCP (currently)
DDL
Must be a clustered index on every table
No physical file placement
No indexed views
No “not for replication” constraint allowed
No Extended properties
Some index options missing (e.g. allow_row_locks, sort_in_tempdb ..)
No set ansi_nulls on

Types
No spatial or hierarchy id
No Text/images support.
Use nvarchar(max)
XML datatype and schema allowed but no XML index or schema collection.
Security
No integrated security

Development
No CLR
Local temp tables are allowed
Global temp tables are not allowed
Cannot alter database inside a connection
No UDDT’s
No ROWGUIDCOL column property

Lessons LearnedSQL Azure vs Windows Azure Tables
SQL Server is very familiar
SQL Azure *is* SQL Server in the cloud
Windows Azure Storage is…very different 
Make the right choice
Understand Azure storage
Understand SQL Azure
Understand they are totally different
You can use both

Lessons Learned SQL Azure vs Windows Azure Tables
SQL Azure is not always the best storage option
SQL Azure costs more
Delivers a *lot* more functionality
SQL Azure is more limited on scale

Lessons Learned SQL Azure and Sharding
Can be done
Many 10GB databases
Not fun 

Queues

Simple asynchronous dispatch queue
Create and delete queues
Message:
Retrieved at least once
Max size 8kb
Operations:
Enqueue
Dequeue
RemoveMessage
Queues

Using the Cloud for Communications
http://app.queue.core.windows.net/
Azure Queue
REST
Client

Company 1
Client
Azure Queue
REST
Company 2
Client

x
Company 1
Client
Azure Queue
REST
Company 2
Client

Company 1
Client
Azure Queue
REST
Web Role
Company 2
Client

In Summary

Windows Azure Platform Benefits
Windows Azure
High Level of Abstraction
Hardware
Server OS
Network Infrastructure
Web Server
Availability
Automated Service Management
Scalability
Instance & Partitions
Developer Experience
Familiar Developer Tools
SQL Azure
Higher Level of Abstraction
Hardware
Server OS
Network Infrastructure
Database Server
Availability
Automated Database Management & Replication
Scalability
Databases Partitioning
Developer Experience
Familiar SQL Environment

Resources
Slides, links and more
http://geekswithblogs.net/iupdateable
Azure Training Kit (August update)
www.azure.com
Sign up, links to resources etc
http://www.azureadvantage.co.uk/
Rapid provisioning of Windows Azure

http://geekswithblogs.net	229
http://www.slideshare.net	37
http://cc.bingj.com	2
http://gyanmoti.wordpress.com	1

Design Considerations For Storing With Windows Azure

by Eric Nelson on Sep 24, 2009

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 269

Statistics

Design Considerations For Storing With Windows Azure — Presentation Transcript