HYBRID SOLUTIONS ENABLEDCloud On Your Terms
THE FANTASTIC 12 OF 2012        1         2        3    4        5         6        7    8        9        10        11   12
SQL DATA SYNC OVERVIEW
SQL SERVER DATA SYNC—KEY FEATURES
PART 1—SETTING UP THE SYNC GROUP
PART 2—EXPANDING THE SYNC GROUP
RESOURCES
LONG PATH TO DATA…
CONNECT WITH THE WORLD’S DATAWindows Azure Marketplacehttps://datamarket.azure.com
DATAMARKETWindows Azure marketplace
WINDOWS AZURE MARKETPLACEBusiness ModelBase meter: $ per transaction (query or API call), charged per transaction or month...
WINDOWS AZURE MARKETPLACE MOMENTUMRegistered Users: ~50,000 (double digit growthmonth over month) (38 countries)Subscripti...
DELIVERING DATA THROUGH MICROSOFT’S CHANNELS
HOW TO ACCESS THE DATA?Access from any platform or any app:Native OData-based APIs support Query language over HTTP Open s...
WHY DATA MOVEMENT IS SO CRITICAL IN AZURE?It is a common requirement for applications running in the cloudMay have impact ...
BULK COPY PROGRAMFast option to move data in and out from Azure SQL DatabaseCan run on premises, in a web/worker role or i...
SQLBULKINSERT CLASS IN ADO.NETProvide a way to programmatically move data into SQLDatabases from .NET codeWrapper around b...
SQLBULKINSERT CLASS IN ADO.NETCan be easily integrated with components like TransientFault Handling App Block to provide r...
ADDITIONAL CONSIDERATION ON BATCHING DATA INSERTSExecuting multiple insert operations in a single roundtrip is a way tomin...
ADDITIONAL CONSIDERATION ON BATCHING DATA INSERTSUsing a Table Valued Parameter approach in a “INSERTINTO…. SELECT FROM @T...
SQL SERVER INTEGRATION SERVICESRich and mature data transformation framework and pipelineWell known to traditional DBAs an...
BACPAC FORMAT AND IMPORT/EXPORT SERVICEBacpac format provides a way to logically represent data models,database objects (t...
BACPAC FORMAT AND IMPORT/EXPORT SERVICEImport/Export service doesnt provide a transactionally consistent copy of yourSQL D...
GRADE OF THE STEEL: AZURE DATA MOVEMove data fast:Minimize downtime window finding thefastest and more reliable data movem...
KEY DATA MOVEMENT SCENARIOSInitial data loading and migrationData synchronizationBackup/restore for data protection and di...
INITIAL DATA LOADING AND MIGRATIONMove existing on premises relational data to Azure SQL DatabasesMay reside in SQL Server...
DATA SYNCHRONIZATIONAfter an initial data loading, continuous or scheduled synchronizationactivities between on premises d...
MOVING CLOUD GENERATED DATA TO ON PREMISESThe ability to move cloud generated data from Azure SQL Databases toon premises ...
AZURE SQL DATABASE BACKUP / DR OPTIONSBackup/restore for data protection and disaster recovery“Whoops” errors protection a...
SHARDING / RE-SHARDINGTake an existing monolithic database and move data into asharded/partitioned data tierInitial shardi...
GENERAL SUGGESTIONS
© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be...
Sql server 2012_sp1_07_of_12_hybrid_solutions_enabled_level300_dark
Sql server 2012_sp1_07_of_12_hybrid_solutions_enabled_level300_dark
Sql server 2012_sp1_07_of_12_hybrid_solutions_enabled_level300_dark
Sql server 2012_sp1_07_of_12_hybrid_solutions_enabled_level300_dark
Sql server 2012_sp1_07_of_12_hybrid_solutions_enabled_level300_dark
Sql server 2012_sp1_07_of_12_hybrid_solutions_enabled_level300_dark
Sql server 2012_sp1_07_of_12_hybrid_solutions_enabled_level300_dark
Sql server 2012_sp1_07_of_12_hybrid_solutions_enabled_level300_dark
Sql server 2012_sp1_07_of_12_hybrid_solutions_enabled_level300_dark
Upcoming SlideShare
Loading in …5
×

Sql server 2012_sp1_07_of_12_hybrid_solutions_enabled_level300_dark

272 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Sql server 2012_sp1_07_of_12_hybrid_solutions_enabled_level300_dark

  1. 1. HYBRID SOLUTIONS ENABLEDCloud On Your Terms
  2. 2. THE FANTASTIC 12 OF 2012 1 2 3 4 5 6 7 8 9 10 11 12
  3. 3. SQL DATA SYNC OVERVIEW
  4. 4. SQL SERVER DATA SYNC—KEY FEATURES
  5. 5. PART 1—SETTING UP THE SYNC GROUP
  6. 6. PART 2—EXPANDING THE SYNC GROUP
  7. 7. RESOURCES
  8. 8. LONG PATH TO DATA…
  9. 9. CONNECT WITH THE WORLD’S DATAWindows Azure Marketplacehttps://datamarket.azure.com
  10. 10. DATAMARKETWindows Azure marketplace
  11. 11. WINDOWS AZURE MARKETPLACEBusiness ModelBase meter: $ per transaction (query or API call), charged per transaction or monthly subscriptions 1. Trial Usage: Try before you buy 2. Per Transaction: Pay-as-you-grow transaction based tiers 3. Monthly Subscription: Subscribe to the dataset for unrestricted accessRevenue sharing MS(20%) and content providers (80%)Free offers – no cost
  12. 12. WINDOWS AZURE MARKETPLACE MOMENTUMRegistered Users: ~50,000 (double digit growthmonth over month) (38 countries)Subscriptions: 70,000+ (double digit growth monthover month)Providers: 500+Datasets: 130+ across 15 categories, Apps: 600+Data Quality Services, Microsoft Translator, BingSearch API
  13. 13. DELIVERING DATA THROUGH MICROSOFT’S CHANNELS
  14. 14. HOW TO ACCESS THE DATA?Access from any platform or any app:Native OData-based APIs support Query language over HTTP Open standard protocol (www.odata.org) Native service reference usage in Visual Studio Downloadable proxy classes for fixed function servicesURL-based query supportAuthentication and authorization:Live ID for marketplace access HTTP Basic auth over SSL with account key
  15. 15. WHY DATA MOVEMENT IS SO CRITICAL IN AZURE?It is a common requirement for applications running in the cloudMay have impact onInitial production ramp up (on premises to Azure)Ongoing operations (Azure to Azure)Data protection and recovery (Azure to Azure, Azure to on premises)Data lifecycle management, archiving, pruning (Azure to Azure, Azure to on premises)Moving data between on premises and cloud, and between cloudservices is slower*
  16. 16. BULK COPY PROGRAMFast option to move data in and out from Azure SQL DatabaseCan run on premises, in a web/worker role or in a Virtual machineIf used from on premises, expect potential latency issuesMinimum deployment requirements (SNAC)Can be easily automated, scripted or part of a batch programCan read/write data to/from a local disk, or from an attached persisted .vhd,consider IOPs performance requirements when choosing between the variousoptionsCan run parallel import/export streams from/to the same set of objects toreduce time
  17. 17. SQLBULKINSERT CLASS IN ADO.NETProvide a way to programmatically move data into SQLDatabases from .NET codeWrapper around bulk insert APIsCan run on premises, in a web/worker role or in a VM, butlike other options proximity with the target service willprovide the best performanceAvailable since forever in ADO.NET
  18. 18. SQLBULKINSERT CLASS IN ADO.NETCan be easily integrated with components like TransientFault Handling App Block to provide robust connectionmanagement and support throttling or other issuesProvides all the other benefits of BCP.EXE, like the ability todefine batch size and to run multiple insert streams inparallel
  19. 19. ADDITIONAL CONSIDERATION ON BATCHING DATA INSERTSExecuting multiple insert operations in a single roundtrip is a way tominimize the increased network latency between Windows Azurecompute nodes and SQL Azure databasesBulk Copy APIs provide best performance results over 100 rows inserted,while produce a significant overhead (3.5x) when used with a single rowinsertionBatching multiple INSERT statements in a single Command Text providedgood performance between 1 and 10 records, while produce animportant overhead going to 100 or 1000 records inserted in a singleroundtrip (more than 3x)
  20. 20. ADDITIONAL CONSIDERATION ON BATCHING DATA INSERTSUsing a Table Valued Parameter approach in a “INSERTINTO…. SELECT FROM @TVP” query statement providethe consistently good result results in batches of 1-10-100-1000 rows per roundtrip
  21. 21. SQL SERVER INTEGRATION SERVICESRich and mature data transformation framework and pipelineWell known to traditional DBAs and DevsMany useful standard tasks and capabilities included out of the boxOfficially supported in WA VMsIt works on Web/Worker Roles, but not supportedAutomated package creation and deployment makes SSIS aninteresting component to implement complex data movementsolution in a hybrid scenario
  22. 22. BACPAC FORMAT AND IMPORT/EXPORT SERVICEBacpac format provides a way to logically represent data models,database objects (t-sql code, indexes,etc) with version support and otherbenefits, together with a JSON serialized instance of database rows.The Azure SQL Database Import/Export service uses the Bacpac format tomove data from/to a database and the Azure Blob Storage from themanagement portal or a scriptable service endpoint.Bacpac files can be then easily downloaded on premises and importedinto a SQL Server instance, or being manipulated through aprogrammatic API.
  23. 23. BACPAC FORMAT AND IMPORT/EXPORT SERVICEImport/Export service doesnt provide a transactionally consistent copy of yourSQL Database, so you can either stop users workload or execute thisoperation n a copy of your database executed with the ACREATE DATABASEAS COPY OF option.This option may seems the slowest one, compared to a raw export/importoperation, but youn need to consider that it performs several important tasksthat youll need to execute anyway, like rebuilding a complex indexingstrategy, compress exported data and copy it t the Azure storage.If your data movement needs involve some of these operations,Import/Export service may be your option.
  24. 24. GRADE OF THE STEEL: AZURE DATA MOVEMove data fast:Minimize downtime window finding thefastest and more reliable data movementmechanism availableWarning: based on previous assumptionsthis process is non-deterministic innatureMove Data Reliably:May fail in between move phases as wellCheck for consistency and implementretry logic is necessary
  25. 25. KEY DATA MOVEMENT SCENARIOSInitial data loading and migrationData synchronizationBackup/restore for data protection and disaster recovery“Whoops” errors protection and recoverySharding / Re-Sharding
  26. 26. INITIAL DATA LOADING AND MIGRATIONMove existing on premises relational data to Azure SQL DatabasesMay reside in SQL Server or in a competitive productMany existing tools available for this taskImport/Export Service, BCP.EXE, SSISRecommended approach is to export data out of the on premisesdatabases, compress and copy to a shared storage on Azure, andrun your import operations from Azure roles to limit latency impactParallel loading streams can help performance
  27. 27. DATA SYNCHRONIZATIONAfter an initial data loading, continuous or scheduled synchronizationactivities between on premises databases and cloud may be requiredData Sync Service is a cloud based solution, build on top of the Synch-FX,that can hide most of the complexities and provide a code less solutiondedicated to DBAs and system engineersDepending on the complexity of the sync flow, custom solution leveragingchange data capture on premises or other differencing approaches canbe appliedThis scenario is critical for Hybrid Cloud implementations
  28. 28. MOVING CLOUD GENERATED DATA TO ON PREMISESThe ability to move cloud generated data from Azure SQL Databases toon premises systems can be seen as a mono-directional version of datasynchronizationDepending on the complexity of customer requirements, this can beimplemented by taking a “snapshot” of the production database, movingthis into the Azure Storage, to be downloaded on premises and loadedinto a traditional system (e.g. a data warehousing system)If customer requirements will increase, a more invasive approach could bebased on differencing data movement, limited only to the data that havebeen modified over time
  29. 29. AZURE SQL DATABASE BACKUP / DR OPTIONSBackup/restore for data protection and disaster recovery“Whoops” errors protection and recoveryPITR – Point In Time Restore (Private CTP)14 days rolling database backups automatically, in addition to local 3 or 4 replicas per dbCurrently: restores the entire physical DB + all logical dbs, then extracts your logical dbDBCopyLimited scaling optionsMust complete within 24 hoursDAC Import/Export ServicesService runs in worker rolesScale by adding more roles to support large number of db backups per hour~100K dbs, avg ~300MB each in 6 hours using about 200 small instancesBackup file will be geo-replicated (Windows Azure Blob Storage)NOTE: It does not guarantee transactional consistency – how to solve?
  30. 30. SHARDING / RE-SHARDINGTake an existing monolithic database and move data into asharded/partitioned data tierInitial sharding operation, source database can be on premises or already in AzureSQL DatabasesSplit an existing shard into one or more destination databases forscaling out or data management purposesUsually a cloud to cloud operation, may become a recurring one based on the applicationand workload requirementsCould be ideally performed online, but if the application cantolerate some downtime it will be much easier to implement
  31. 31. GENERAL SUGGESTIONS
  32. 32. © 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademar ks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accu racy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

×