Data Storage inWindows AzureBlaž Lipušček,Diventic d.o.o.                  Nivo 300
Cloud ComputingComputing capability delivered as a utility throughInternet standards and protocols.• Software as a Service...
Windows Azure Fabric• Fabric Controller is the brain of the entire ecosystem• Highly redundant hardware and software envir...
Windows Azure Storage Options • Instance local storage is not persistent:    • Usefull for cache or temp files! • Blob ser...
Windows Azure Blob Service • REST API                             Nivo 300
Store Files to Blob service • WindowsAzure.StorageClient (REST API)                                           Nivo 300
Grant Access to Files • Shared Access Signatures                                                       • Read             ...
Grant Access to Files:Shared Access Signatures • Remember to use UTC on DateTime • Azure hides resources with invalid Shar...
Windows Azure Table Storage • http://myaccount.table.core.windows.net                                             Nivo 300
Windows Azure Table Storage • Requires major changes to application data logic • No Schema, NO-SQL • More scalable than SQ...
Windows Azure Tables • Table names must conform to these rules:    ^[A-Za-z][A-Za-z0-9]{2,62}$ • An entity can have up to ...
Windows Azure Tables • Not allowed in PartitionKey and RowKey values: /, , #, ? • The partition and row key is a string va...
Windows Azure Tables • REST service via HTTP(S):    • CREATE:       http://myaccount.table.core.windows.net/Tables    • LI...
Query Timeout and Pagination • Query can return up to 1,000 rows at a time:    •   x-ms-continuation-NextTableName    •   ...
Query Timeout and PaginationDate: Mon, 13 Feb 2012 20:11:08 GMTContent-Type: application/atom+xml;charset=utf-8Server: Win...
Query OptionsOperator             URI expressionEqual                eqGreaterThan          gtGreaterThanOrEqual   geLessT...
Windows Azure Table Storage  PartitionKey   RowKey     Timestamp   Property Collection                                    ...
Windows Azure Table StorageDate Type    DescriptionString       A UTF-16-encoded string of characters up to 64 KB in size....
DataServiceContext • It is not stateless (like WCF Data Services) • Any instance members are not guaranteed to be thread s...
Table Storage: Good to know! • DataServiceContext is not stateless!     • Create it when you need to perform multiple tran...
Retrieve a Single Entity • Fastest query • Handle 404 Not Found                           Nivo 300
Retrieve Using Prefix                        Nivo 300
Entity Group Transactions  • Limited to a single partition in a single table  • context.SaveChanges(               SaveCha...
Parallel Querying for Full Table Scans • Fetch smaller list of partition keys (eg. Categories) • Execute multiple queries ...
Use Command Pattern   TableCommand (abstract)  Insert   Update     Delete                               Nivo 300
TableCommand abstract class                              Nivo 300
Select Command                 Nivo 300
Update Command                 Nivo 300
Update Command                 Nivo 300
Fat Entities • Instead of going for 64KB limitation per property > single 1MB     limitation for the whole entity. • Table...
Fat Entities               Nivo 300
Table Storage PerformanceOptimizationUse when table and entity name aredifferenct. 8-15ms for deserializingeach entity rec...
Windows Azure Storage Analytics • Track, Analyze, and Debug usage of Windows Azure Storage • Logs: Provide trace of execut...
Windows Azure Storage Analytics • Transaction data and capacity data is stored in the following tables:     • $MetricsCapa...
Logs •   Loggin Authenticated Requests:      • Successful requests      • Failed requests, including timeout, throttling, ...
Metrics • Request information: Provides hourly aggregates of number of    requests, average server side latency, average E...
Get/Set Storage Service PropertiesElement Name              DescriptionVersion                   Required. The version of ...
Windows Azure StorageAnalytics Tools• Cerebrata WAS Configuration Utility  https://www.cerebrata.com/Downloads/SACU/SACU.a...
Connecting to SQL Azure • Connecting to SQL Azure only requires change in configuration • No support for Windows Authentic...
Connecting to SQL AzureOpen connections late and close early. • A logged-in session that has been idle for 30 minutes will...
SQL Azure Feature Limitations • Common Language Runtime (CLR) and CLR User-Defined Types • Integrated Full-Text Search • S...
Troubleshooting SQL Azure •   Service Unavailable or Does Not Exist     • Check if you are unable to resolve the name     ...
SQL Azure vs Table StorageFeature                 SQL Azure                     Common Benefit(s)             Azure Table ...
Windows Azure Cost structureIn the context of the cost-oriented architecture, variousdecisions will impact monthly operati...
Windows Azure PricingWindows Azure Capability      ChargeServer Usage                  Small: $0.12 /service-hour         ...
VPRAŠANJA?Po zaključku predavanja, prosimo, izpolnite vprašalnik.Vprašalniki bodo poslani na vaš e-naslov, dostopni pabodo...
Upcoming SlideShare
Loading in...5
×

Data Storage In Windows Azure Ntk 2012

1,274

Published on

Na predavanju bomo spoznali tehnologije Windows Azure Blob in Table Storage, ki omogočata shranjevanje in obdelavo velikih količin podatkov v oblaku. Predstavljeni bodo primeri in najboljše prakse uporabe omenjenih tehnologij, ter njihove prednosti in omejtive. Pogledali bomo, kako se SQL Azure razlikuje od Table Storage.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,274
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Data Storage In Windows Azure Ntk 2012

  1. 1. Data Storage inWindows AzureBlaž Lipušček,Diventic d.o.o. Nivo 300
  2. 2. Cloud ComputingComputing capability delivered as a utility throughInternet standards and protocols.• Software as a Service• Platform as a Service• Infrastructure as a Service Nivo 300
  3. 3. Windows Azure Fabric• Fabric Controller is the brain of the entire ecosystem• Highly redundant hardware and software environment• It can‘t ever fail! Nivo 300
  4. 4. Windows Azure Storage Options • Instance local storage is not persistent: • Usefull for cache or temp files! • Blob service: • Store large text or binary data • Files can be served via CDN • Table service • Azure Drive: • NTFS, can be used with I/O • Can connect only one instance at once Nivo 300
  5. 5. Windows Azure Blob Service • REST API Nivo 300
  6. 6. Store Files to Blob service • WindowsAzure.StorageClient (REST API) Nivo 300
  7. 7. Grant Access to Files • Shared Access Signatures • Read • Write • Delete • List https://mycontainer.blob.core.windows.net/files/image.jpg?se=2011-05- 18T12%3A50%3A38Z&sr=b&sp=r&sig=EUtAbKQ%2F%2FYY7k1fJtzE6hWt%2FjadF AqxDYh5tAPK%2FeIw%3D Nivo 300
  8. 8. Grant Access to Files:Shared Access Signatures • Remember to use UTC on DateTime • Azure hides resources with invalid Shared Access Signatures • DeleteIfExists will never fail • Make sure URL is properly encoded Nivo 300
  9. 9. Windows Azure Table Storage • http://myaccount.table.core.windows.net Nivo 300
  10. 10. Windows Azure Table Storage • Requires major changes to application data logic • No Schema, NO-SQL • More scalable than SQL Azure • Can handle large datasets • Relatively cheap Nivo 300
  11. 11. Windows Azure Tables • Table names must conform to these rules: ^[A-Za-z][A-Za-z0-9]{2,62}$ • An entity can have up to 255 properties • The combined size of all data in an entitys cannot exceed 1 MB • 64KB limitation per property • An entity always has the following system properties: • PartitionKey, RowKey,Timestamp property Nivo 300
  12. 12. Windows Azure Tables • Not allowed in PartitionKey and RowKey values: /, , #, ? • The partition and row key is a string value up to 1 KB in size. • You must include the PartitionKey and RowKey property in every insert, update, and delete operation. Nivo 300
  13. 13. Windows Azure Tables • REST service via HTTP(S): • CREATE: http://myaccount.table.core.windows.net/Tables • LIST: http://myaccount.table.core.windows.net/Tables • GET, UPDATE, DELETE: http://myaccount.table.core.windows.net/MyTable() • SINGLE TABLE: http://myaccount.table.core.windows.net/Tables(MyTable) Nivo 300
  14. 14. Query Timeout and Pagination • Query can return up to 1,000 rows at a time: • x-ms-continuation-NextTableName • x-ms-continuation-NextPartitionKey • x-ms-continuation-NextRowKey • Construct new query using continuation tokens: • NextTableName, NextPartitionKey , NextRowKey • 30 seconds for scheduling and processing Nivo 300
  15. 15. Query Timeout and PaginationDate: Mon, 13 Feb 2012 20:11:08 GMTContent-Type: application/atom+xml;charset=utf-8Server: Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0Cache-Control: no-cachex-ms-request-id: f9b2cd09-4dec-4570-b06d-4fa30179a58ex-ms-version: 2011-08-18x-ms-continuation-NextPartitionKey: 1!8!U21pdGg-x-ms-continuation-NextRowKey: 1!12!QmVuMTg5OA--Content-Length: 880298 http://myaccount.table.core.windows.net/Customers? NextPartitionKey=1!8!U21pdGg-& NextRowKey=1!12!QmVuMTg5OA-- Nivo 300
  16. 16. Query OptionsOperator URI expressionEqual eqGreaterThan gtGreaterThanOrEqual geLessThan ltLessThanOrEqual le Operator URI expressionNotEqual ne And and Not not Or or Nivo 300
  17. 17. Windows Azure Table Storage PartitionKey RowKey Timestamp Property Collection Name Type Value Jeff C_Jeff DateTime Kind String “Customer” Name String “Jeffrey” City String “Seattle” Jeff O_Cereal DateTime Kind String “Order” Item String “Cereal” Quantity Int32 1 Jeff O_Milk DateTime Kind String “Order” Item String “Milk” Quantity Int32 2 Paul C_Paul DateTime Kind String “Customer” Name String “Paul” City String “New York” Nivo 300
  18. 18. Windows Azure Table StorageDate Type DescriptionString A UTF-16-encoded string of characters up to 64 KB in size.Byte array An array of bytes up to 64 KB in size.Guid A 128-bit globally unique identifier (can be null).DateTime A 64-bit value expressed as Coordinated Universal Time (UTC) ranging from 12:00 midnight, January 1, 1601 A.D. (C.E.) to December 31, 9999 (can be null).Int32 A 32-bit signed integer (can be null)Int64 A 64-bit signed integer (can be null).Double A 64-bit floating point value (can be null)Boolean A Boolean value (can be null) Nivo 300
  19. 19. DataServiceContext • It is not stateless (like WCF Data Services) • Any instance members are not guaranteed to be thread safe Nivo 300
  20. 20. Table Storage: Good to know! • DataServiceContext is not stateless! • Create it when you need to perform multiple transactions • Set timeout by using the Timeout property • Query using PartitionKey and RowKey • Only this two parameters are indexed. • Order records by latest: • DateTime.MaxValue.Ticks - date.ToUniversalTime().Ticks Nivo 300
  21. 21. Retrieve a Single Entity • Fastest query • Handle 404 Not Found Nivo 300
  22. 22. Retrieve Using Prefix Nivo 300
  23. 23. Entity Group Transactions • Limited to a single partition in a single table • context.SaveChanges( SaveChangesOptions.Batch);PartitionKey RowKey ID Kind Amount Title ... Timestamp7635 7635 7635 ExpenseItem MyExpense 15.5...30:007635 7635_8215 8215 ExpenseItemRow 100 15.5...30:057635 7635_8216 8216 ExpenseItemRow 50 15.5...30:10 Nivo 300
  24. 24. Parallel Querying for Full Table Scans • Fetch smaller list of partition keys (eg. Categories) • Execute multiple queries in parallel: • serviceContext.FoodTable.AsTableServiceQuery() .Where(c => c.PartitionKey == _category && c.Color == "Green"); • Parallel LINQ Nivo 300
  25. 25. Use Command Pattern TableCommand (abstract) Insert Update Delete Nivo 300
  26. 26. TableCommand abstract class Nivo 300
  27. 27. Select Command Nivo 300
  28. 28. Update Command Nivo 300
  29. 29. Update Command Nivo 300
  30. 30. Fat Entities • Instead of going for 64KB limitation per property > single 1MB limitation for the whole entity. • Table provider also abstracts away all subtle retry policies: • Handles OperationTimedOut error • 4MB restriction of the Table Storage for transactions • http://code.google.com/p/lokad-cloud/wiki/FatEntities Nivo 300
  31. 31. Fat Entities Nivo 300
  32. 32. Table Storage PerformanceOptimizationUse when table and entity name aredifferenct. 8-15ms for deserializingeach entity received in a query. Payload delay. Overhead with POST and PUT requests. Nivo 300
  33. 33. Windows Azure Storage Analytics • Track, Analyze, and Debug usage of Windows Azure Storage • Logs: Provide trace of executed requests for Blobs, Tables and Queues • Metrics: Provide summary of key capacity and request statistics for Blobs, Tables and Queues • Data stored inside your account • Turn it on with Get Storage Service Properties API call Nivo 300
  34. 34. Windows Azure Storage Analytics • Transaction data and capacity data is stored in the following tables: • $MetricsCapacityBlob • $MetricsTransactionsBlob • $MetricsTransactionsTable • $MetricsTransactionsQueue Nivo 300
  35. 35. Logs • Loggin Authenticated Requests: • Successful requests • Failed requests, including timeout, throttling, network, authorization ... • Requests using a SAS, including failed and successful requests • Requests to analytics data • Loggin Anonymous Requests: • Successful requests • Server errors • Timeout errors for both client and server • Failed GET requests with error code 304 (Not Modified) Nivo 300
  36. 36. Metrics • Request information: Provides hourly aggregates of number of requests, average server side latency, average E2E latency, average bandwidth, total successful requests and total number of failures and more. • Capacity information: Provides daily statistics for the space consumed by the service, number of containers and number of objects that are stored in the service. Note, this is currently only provided for the Windows Azure Blob service. Nivo 300
  37. 37. Get/Set Storage Service PropertiesElement Name DescriptionVersion Required. The version of Storage Analytics to configure.Delete Required. Applies only to logging configuration. Indicates whether all delete requests should be logged.Read Required. Applies only to logging configuration. Indicates whether all read requests should be logged.Write Required. Applies only to logging configuration. Indicates whether all write requests should be logged.Enabled Required. Indicates whether metrics is enabled for the storage service.IncludeAPIs Required only if metrics is enabled. Applies only to metrics configuration. Indicates whether metrics should generate summary statistics for called API operations.RetentionPolicy/Enabled Required. Indicates whether a retention policy is enabled for the storage service.RetentionPolicy/Days Required only if a retention policy is enabled. Indicates the number of days that metrics or logging data should be retained. All data older than this value will be deleted. The minimum value you can specify is 1; the largest value is 365 (one year). Nivo 300
  38. 38. Windows Azure StorageAnalytics Tools• Cerebrata WAS Configuration Utility https://www.cerebrata.com/Downloads/SACU/SACU.application• Sample Tool http://storageanalytics.cloudapp.net• Cerebrata Cloud Storage Studio• Cerebrata Diagnostics Manager Nivo 300
  39. 39. Connecting to SQL Azure • Connecting to SQL Azure only requires change in configuration • No support for Windows Authentication • Encrypt credentials Nivo 300
  40. 40. Connecting to SQL AzureOpen connections late and close early. • A logged-in session that has been idle for 30 minutes will be terminated automatically. Nivo 300
  41. 41. SQL Azure Feature Limitations • Common Language Runtime (CLR) and CLR User-Defined Types • Integrated Full-Text Search • SQL Server 2008 R2 Features (SQL Server Utility, PowerShell Provider, Master Data Services) • SQL Server Replications, Backup and Restore CREATE DATABASE destination_database_name AS COPY OF [source_server_name.]source_database_name Nivo 300
  42. 42. Troubleshooting SQL Azure • Service Unavailable or Does Not Exist • Check if you are unable to resolve the name • Firewall blocking port 1433 used by SQL Azure • Using proxy server not configured properly • General Network Errors • Connection is idle for an extended period of time. • Connection consumes an excessive amount of resources • Connection holds onto a transaction for an extended period of time • If the server is too busy https://www.windowsazure.com/en-us/support/service-dashboard/ Nivo 300
  43. 43. SQL Azure vs Table StorageFeature SQL Azure Common Benefit(s) Azure Table Storage Cross-table queries Primary key-based lookup Single key lookup (bySelect semantics partition) High performance via Automatic mass scale by multiple indices, partition and consistentPerformance and scale normalized data structures. performance even at large scale Well-known management Familiar high-level Direct serialization;User experience tools and traditional developer experience simplified design model by database design removing relational model Traditional relational design Data storage for all types of Type variability in a singleStorage style model data table No transaction cost, pay by Network traffic cost outside No space overhead cost,Cost factors database size of same datacenter pay for what is used Synchronizing between local and cloud-basedData loading and sync stores; data easily moved in and out. Nivo 300
  44. 44. Windows Azure Cost structureIn the context of the cost-oriented architecture, variousdecisions will impact monthly operational expenses.• Compute charges Compute• Storage charges Storage• Charges for Data Transfer Transfer Services• Add-on service charges Nivo 300
  45. 45. Windows Azure PricingWindows Azure Capability ChargeServer Usage Small: $0.12 /service-hour Medium: $0.24/service-hour Large: $0.48/service-hour XLarge: $0.96/service-hourWindows Azure Blobs and $0.14/GBTablesTransactions $0.01/10K transactionsSQL Azure: Web Edition $9.99/month (1GB RDBMS)SQL Azure: Business Edition $99.99/month (10GB RDBMS)Windows Azure $0.15/100K message operations Nivo 300
  46. 46. VPRAŠANJA?Po zaključku predavanja, prosimo, izpolnite vprašalnik.Vprašalniki bodo poslani na vaš e-naslov, dostopni pabodo tudi preko profila na spletnem portalu konferencewww.ntk.si.Najlepša hvala!Blaž Lipušček, Diventic d.o.o.blaz.lipuscek@diventic.si Nivo 300
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×