Microsoft Database Options


Published on

Joe Shirey's deck delivered at the 2009Q4 Microsoft Architect Council events

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Microsoft Database Options

  1. 1. Data Options in the Cloud<br />Joe Shirey<br /><br />
  2. 2. Database Choices<br />Value Props:<br />Full h/w control – size/scale<br />100% of API surface area<br />Roll-your-own HA/DR/scale<br />Value Props:<br />100% of API surface area<br />Roll-your-own HA/DR/scale<br />Dedicated<br />On-premise<br /> SQL Server or other s/w on-premise<br /> Resource governance @ machine<br /> Security @ DB Server/OS<br />Value Props:<br />Auto HA, Fault-Tolerance<br />Friction-free scale<br />Self-provisioning<br />Subset of API surface area<br />Resources<br />Hosted<br /> Hosted SQL Server<br /> Resource governance @ VM<br /> Security @ DB Server/OS<br />SQL Azure Database<br /> Virtual DB server<br />Resource governance @ DB<br /> Security @ DB/Virtual Server<br />Shared<br />Low<br />“Friction”/Control<br />High<br />SQL Azure V1 targets scenarios that live in the lower left quadrant<br />
  3. 3. SQL Server 2008 R2<br />Focus Areas<br />Improvements in Scalability and Performance<br />Enhanced Manageability<br />Self Service Business Intelligence<br />Master Data Management<br />
  4. 4. Master Data Management<br />Single source of truth for Master Data<br />Thin client management<br />Safeguards of data integrity<br />Data versioning<br />Import/Export capabilities<br />
  5. 5. SQL Server Versions<br />New with SQL 2008 R2<br /><ul><li>Datacenter Edition (256 logical processor support)
  6. 6. Parallel Data Warehouse Edition (100+ TB support)</li></li></ul><li>Cloud Development<br />WEB & CLOUDS<br />third party cloud<br />web applications<br />developer experience<br />use existing skills and tools<br />compute<br />storage<br />management<br />management<br />relational data<br />connectivity<br />access control<br />ON-PREMISES<br />lob Applications<br />composite applications<br />
  7. 7. Storage in the Cloud<br />WEB & CLOUDS<br />third party cloud<br />web applications<br />developer experience<br />use existing skills and tools<br />management<br />management<br />connectivity<br />access control<br />compute<br />relational data<br />storage<br />ON-PREMISES<br />lob Applications<br />composite applications<br />
  8. 8. Why Data in the Cloud<br />Capital Costs<br />DBA (or lack thereof)<br />High Availability<br />Performance<br />Scalability <br />
  9. 9. Windows Azure Storage<br />The goal is to allow users and services<br />Anywhere at anytime access<br />Store data for any length of time <br />Scale to store any amount of data<br />Be confident that the data will not be lost <br />Pay for only what they use/store<br />
  10. 10. Windows Azure Storage<br />Storage<br />Durable<br />Scalable (capacity and throughput)<br />Highly available<br />Rich storage concepts<br />Large user data items: blobs<br />Service state: tables<br />Service communication: queues<br />Simple and familiar programming interfaces<br />REST (HTTP and HTTPS)<br />.NET accessible<br />
  11. 11. Fundamental data abstractions<br />Blobs – provide a simple interface for storing named files along with metadata for the file<br />Tables – provide structured storage. a table is a set of entities, which contain a set of properties<br />Queues – provide reliable storage and delivery of messages for an application<br />
  12. 12. Blob Storage Concepts<br />blob<br />container<br />account<br />IMG001.JPG<br />pictures<br />MOV1.AVI<br />IMG002.JPG<br />sally<br />movies<br />
  13. 13. Blob Features and Functions<br />Store large objects (up to 200GB)<br />Associate metadata with blob<br />metadata is &lt;name, value&gt; pairs, up to 8KB per blob<br />set/get with or separate from blob data bits<br />Standard REST Interface<br />PutBlob<br />Inserts a new blob or overwrites the existing blob<br />GetBlob<br />Get whole blob or a specific range<br />DeleteBlob<br />
  14. 14. Windows Azure Tables<br />Provides structured storage<br />Massively scalable tables<br />Billions of entities (rows) and TBs of data<br />Automatically scales across servers as traffic grows<br />Highly available<br />Anywhere at anytime access to your data<br />Durable<br />Data is replicated at least 3 times<br />Familiar and easy to use programming interfaces<br />ADO.NET data services – .NET 3.5 SP1<br />.NET classes and LINQ<br />REST - with any platform or language<br />
  15. 15. Table Storage Concepts<br />entity<br />table<br />account<br />Name =…<br />Email = …<br />users<br />Name =…<br />Email = …<br />sally<br />Photo ID =…<br />Date =…<br />photo index<br />Photo ID =…<br />Date =…<br />
  16. 16. Table Data Model<br />Table<br />A storage account can create many tables<br />Table name is scoped by account<br />Data is stored in tables<br />A table is a set of entities (rows)<br />An entity is a set of properties (columns)<br />Entity<br />Two “key” properties that together are the unique ID of the entity in the table<br />PartitionKey – enables scalability<br />RowKey – uniquely identifies the entity within the partition<br />
  17. 17. Partition Deyand Partitions<br />Every table has a partition key<br />It is the first property (column) of your table<br />Used to group entities in the table into partitions<br />A table partition <br />All entities in a table with the same partition key value<br />Partition key is exposed in the programming model <br />Allows application to control the granularity of the partitions and enable scalability<br />
  18. 18. Partition Example<br />Table partition – all entities in tablewith same partition key value<br />Application controls granularity of partition<br />Partition 1<br />Partition 2<br />
  19. 19. Purpose of the Partition Key<br />Entity locality<br />Entities in the same partition will be stored together<br />Efficient querying and cache locality<br />Entity group transactions <br />Atomically perform multiple insert/update/delete over entities in same partition in a single transaction<br />Table scalability<br />We monitor the usage patterns of partitions<br />Automatically load balance partitions<br />Each partition can be served by a different storage node<br />Scale to meet the traffic needs of your table<br />
  20. 20. Choosing a Partition Key<br />Granularity of entity group transactions<br />Make the partition key only as big as you need it for entity group transactions<br />Spread out load across partitions<br />More partitions – makes it easier to automatically balance load<br />Currently have one primary index<br />Important to use a partition key that is common in your queries<br />If partition key is part of query<br />Fast access to retrieve entities within a single partition<br />If partition key is not specified in a query <br />Then every partition has to be scanned<br />
  21. 21. Table Entities and Properties<br />Each entity can have up to 255 properties<br />Mandatory properties for every entity in table<br />Partition key<br />Row key <br />All entities have a system maintained version<br />No fixed schema for rest of properties<br />Each property is stored as a &lt;name, typed value&gt; pair<br />No schema stored for a table<br />2 entities within the same table can have different properties<br />Properties can be the standard .NET types <br />String, binary, bool, DateTime, GUID, int, int64, and double<br />
  22. 22. Table Programming Model<br />Provide familiar and easy to use interfaces<br />Leverage your .NET expertise<br />Table entities are accessed as objects via ADO.NET Data Services – .NET 3.5 SP1<br />LINQ – language Integrated query<br />RESTful access to table and entities<br />Insert/update/delete entities over the table<br />Query over tables<br />Get back a list of structured entities<br />
  23. 23. Web + Worker Role Pattern<br />Web role<br />Web farm that handles request from the internet<br />Push work items onto storage queue<br />Worker role<br />Process work item off storage queue<br />Public internet<br />n<br />m<br />Web role<br />Worker role<br />Q<br />Load<br /> balancer<br />Cloud storage (tables, blobs, queues)<br />
  24. 24. Windows Azure Queues<br />Provide reliable message delivery<br />Simple, asynchronous work dispatch<br />Programming semantics ensure that a message can be processed at least once<br />Queues are highly available, durable and performance efficient<br />Access is provided via REST<br />
  25. 25. Queue Storage Concepts<br />Message<br />Queue<br />Account<br />128x128, http://…<br />thumbnail jobs<br />256x256, http://…<br />sally<br />http://…<br />photo processing jobs<br />http://…<br />
  26. 26. Account, Queues and Messages<br />An account can create many queues<br />Queue name is scoped by the account<br />Aqueue contains messages<br />No limit on number of messages stored in a queue<br />A message is stored for at most a week in a queue<br />http://&lt;Account&gt;;QueueName&gt;<br />Messages<br />Message size &lt;= 8 KB<br />To store larger data, store data in blob/entity storage, and the blob/entity name in the message<br />
  27. 27. queue programming API<br />queues<br />create/delete/clear queues<br />inspect queue length<br />messages<br />enqueue (queuename, message)<br />dequeue (queueName, invisibility time T)<br />returns the message with a messageID<br />makes the message invisible for time T<br />delete(queuename, messageID)<br />
  28. 28. Queue Best Practices<br />Make message processing idempotent<br />Need to deal with failures<br />No fixed order for dequeue messages<br />Invisible messages result in out of order processing <br />Use the queue length to scale your workers<br />
  29. 29. Demowindows azure storage<br />
  30. 30. SQL Azure<br />SQL Server Data Services false start<br />Familiar relational model<br />Uses existing APIs & tools<br />Friction free provisioning and reduced management<br />Built for the cloud with availability and scale<br />Clear feedback: “I want a SQL database in the cloud”<br />
  31. 31. Service Provisioning Model<br />Each account has zero or more servers<br />Azure wide, provisioned in a common portal<br />Billing instrument<br />Each server has one or more databases<br />Contains metadata about the databases and usage<br />Unit of authentication<br />Unit of geo-location<br />Generated DNS based name<br />Each database has standard SQL objects<br />Unit of consistency<br />Unit of multi-tenancy<br />Contains users, tables, views, indices, etc.<br />Most granular unit of billing<br />SKU’s <br />Web edition -1 GB<br />Business edition – 10 GB<br /> account<br /> server<br /> database<br />
  32. 32. Architecture<br />Shared infrastructure at SQL database and below<br />Request routing, security and isolation<br />Scalable HA technology provides the glue<br />Automatic replication and failover<br />Provisioning, metering and billing infrastructure<br />Machine 5<br />Machine 6<br />Machine 4<br />SQL Instance<br />SQL Instance<br />SQL Instance<br />SQL DB<br />SQL DB<br />SQL DB<br />UserDB1<br />UserDB2<br />UserDB3<br />UserDB4<br />UserDB1<br />UserDB2<br />UserDB3<br />UserDB4<br />UserDB1<br />UserDB2<br />UserDB3<br />UserDB4<br />SDS Provisioning (databases, accounts, roles, …, Metering, and Billing<br />Scalability and Availability: Fabric, Failover, Replication, and Load balancing<br />Scalability and Availability: Fabric, Failover, Replication, and Load balancing<br />
  33. 33. Sample of SQL compatibility<br />In Scope for V1<br />Tables, indexes and views<br />Stored Procedures<br />Triggers<br />Constraints<br />Table variables, session temp tables (#t)<br />…..<br />Out of Scope for V1<br /><ul><li>Distributed Transactions
  34. 34. Distributed Query
  35. 35. CLR
  36. 36. Service Broker
  37. 37. Spatial
  38. 38. Physical server or catalog DDL and views</li></li></ul><li>Logical vs. Physical Administration<br />Customer can focus on logical administration<br />Schema creation and management<br />Query optimization<br />Security management (logins, users, roles)<br />Service handles physical management<br />Automatically replicated with HA “out of the box”<br />Transparent failover in case of failure<br />Load balancing of data to ensure SLA<br />DBA role places more focus on logical management<br />
  39. 39. Programming Model<br />Small data sets<br />Use a single database<br />Same model as on-premise SQL Server<br />Large data sets<br />Partition data across many databases<br />Use parallel fan-out queries to fetch the data<br />Application code must be partition aware<br />
  40. 40. Feature Set Comparison<br />
  41. 41. Management Tooling<br />SQL Server 2008 R2 Management Studio<br />SQLCMD<br />Third party/ communityadd-ons<br />
  42. 42. Data Migration <br />BCP<br />SSIS<br />SQL Azure Migration Wizard<br />
  43. 43. Customer Learning’s from TAP program<br />Use SQL Azure to store metadata and BLOB storage for large files<br />Highly elastic load patterns in a cost effective way is an industry challenge<br />Combination of different SKUs<br />
  44. 44. Sync Framework<br />SQL Azure<br />Local SQL DB<br />Sync Process<br />Sync Process<br />Local Computer<br />MSDN -<br />
  45. 45. DemoSQL Azure storage and Sync<br />
  46. 46. Pricing<br />Windows Azure Storage<br />Bandwidth: $0.10 in / $0.15 out / GB<br />$0.15/GB stored/month<br />SQL Azure Storage<br />Bandwidth: $0.10 in / $0.15 out / GB<br />Web edition (1GB): $9.99/month<br />Business edition (10GB): $99.99/month<br />
  47. 47. Data Options in the Cloud<br /><br /><br />