Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Designing and developing your database for application availability


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Designing and developing your database for application availability

  1. 1. 1Designing and Developing yourDatabase for Application AvailabilityPresented to the Melbourne CBD SQL PASS Chapter.Saxons - Level 8, 500 Collins Street. Melbourne, Australia12:30 – 14:00, July 26th 2010Charley Hanania, QS2 AGB.Sc (Computing), MCP, MCDBA, MCITP, MCTS, MCT, Microsoft MVP: SQL ServerSenior Database Specialist
  2. 2. My Background• Now:• Microsoft MVP: SQL Server• Database Consultant (again, and very happy) at QS2 AG• Formerly:• Production Product Owner of MS SQL Server Platform at UBS Investment Bank• Technical Team Lead• ITIL v3 Certified• SQL Server Certified since 1998• On SQL Server since 1995• Version 4 on OS/2• IT Professional since 1992• PASS• Chapter Leader – Switzerland• Regional Mentor – Europe• European PASS Conference Lead• Event Speaker226 July, 2010Designing&DevelopingDatabasesforApplicationAvailability
  3. 3. Contact Info• Email:• Website: <- Slides here• Twitter:• Blog:• Linked-in: July, 2010Designing&DevelopingDatabasesforApplicationAvailability
  4. 4. Agenda• General Barriers to Application Availability• Standard High Availability Methodology• General Standard - Clustering• One up on the Standard – Database Mirroring• Decreasing component start-up time• Instant File Initialisation• Partitioning your Database Infrastructure• Eureka! Partitioning for Availability (PfA)• Basic PfA Methodology• Code Samples• QuestionsDesigning&DevelopingDatabasesforApplicationAvailability426 July, 2010
  5. 5. General Barriers to AvailabilityDesigning&DevelopingDatabasesforApplicationAvailability626 July, 2010Simplistic Network Topology View:Slightly more realistic view:And that’s just scratching the surface of the Peripherals!Think about the layered complexity of the supporting software and drivers…!Environmental Complexity
  6. 6. Standard HA Methodology• Keep it up for as long as possible.• Increase redundancy• Decrease Single Points of Failure• Bring it back as quickly as possible• Decrease the dependency sequence• Decrease component start-up timeDesigning&DevelopingDatabasesforApplicationAvailability726 July, 2010
  7. 7. General Standard - Clustering• Keep it up for as long as possible.• Increase redundancy• Additional Servers waiting in readiness • Increased redundancy = increased cost • Increased redundancy = decreasedutilisation • Decrease Single Points of Failure• Virtual IP Addresses • Virtual Network Names • Additional Network Cards etc • Storage • Bring it back as quickly as possible• Decrease the dependencysequence• Server is Started • SQL Instance is an exclusive ClusterResource • Decrease component start-up time• Instant File Initialisation • App Database isn’t useable until PrimaryData File is opened and recovered 26 July, 2010Designing&DevelopingDatabasesforApplicationAvailability8PrimaryStandby
  8. 8. Database Mirroring• We could discuss the architecture, benefits and uses of DBMirroring, but I’m told you’re mainly architects anddevelopers, and the doors here aren’t locked…• If you’d like to discuss DB Mirroring, we can chat after thesession…Designing&DevelopingDatabasesforApplicationAvailability926 July, 2010
  9. 9. Decreasing Component Startup Time(Another DBA point, but one that is Golden and not to be missed!)• Data and log files are first initialized by being filled and withzeros when created or restored.• When SQL Server Starts, the tempdb database is recreated.• Application databases do not come online until tempdb has.• If large, then startup times can be quite long!• Action:• Ensure that instant file initialisation is set (Windows:SE_MANAGE_VOLUME_NAME right)• The group that the SQL Server Service Account is in needs to beadded to the Perform Volume Maintenance Tasks security policy,in the “Local Security Policy” Tool.• I’ve seen this reduce SQL Server Startup from 8 minutes to 1second (max)Designing&DevelopingDatabasesforApplicationAvailability1026 July, 2010INSTANT FILE INITIALISATION
  10. 10. Decreasing Component Startup Time• Msdn lists the following (summarised)• Hardware Partitioning• Multiprocessors that enable multiple threads of operations.• RAID (redundant array of independent disks) devices• A table striped across multiple drives can typically be scanned faster thanthe same table stored on a single drive.• Horizontal Partitioning• Divide tables into multiple tables containing the same number ofcolumns, but fewer rows.• Partitioning data horizontally based on age and use is common.• Vertical Partitioning• Divide tables into multiple tables containing fewer columns throughnormalization and row splitting.Designing&DevelopingDatabasesforApplicationAvailability1126 July, 2010PARTITIONING DATABASE COMPONENTS
  11. 11. Additional Partitioning Options• Logical Schema Partitioning• Used mainly for securing and grouping objects by department,role, application etc…• Physical Partitioning• Of Files and Filegroups• Used mainly for performance• Files placed on multiple different data volumes• Multiple files created for round-robin writes and extent allocationsDesigning&DevelopingDatabasesforApplicationAvailability1226 July, 2010
  12. 12. Eureka! Partitioning for Availability• Each database has an area similar to a disk’s boot record• For a database this resides in the Primary data file• If this file is corrupt or unavailable, the database will not beloaded and opened regardless of other data files• A good practice is to not put any objects in this data file, and toput all your tables and objects into secondary data files.• Within the DB you can distribute objects into files to achieveone or more of the following• Enable an application to continue working when other data filesare missing or corrupt by grouping needed objects together• Restore the most critical files for your application system to startup first, allowing partial functionality of your application tocontinue without issue.Designing&DevelopingDatabasesforApplicationAvailability1326 July, 2010
  13. 13. 14Designing&DevelopingDatabasesforApplicationAvailability26 July, 2010Open DiscussionWhich ways could we split the adventureworks db,focussed on client facing / downstream applicationfunctionality, criticality and startup sequence forAvailability?
  14. 14. Adventureworks Schema
  15. 15. PfA Methodology• Conceptual extension from Logical Schema Partitioning andPhysical File Partitioning:1. Don’t put user defined objects in the Primary FileGroupi. Allows the Primary Filegroup to load faster2. Schema Separate user defined objectsi. For clarity of purpose and operational useii. Try not to use dbo schema (good design/operational practice)iii. DBA’s will schematize objects in a similar fashion to what was in theAdventureworks sample. For PfA you should look to map your schema’saround your application’s core functionality and use cases and criticality3. Create Filegroups for each Schemai. Multiple if you want to partition historical data horizontally etcii. Name them in a fashion that allows easy understanding of what isin them, or how they should be used.Designing&DevelopingDatabasesforApplicationAvailability1626 July, 2010
  16. 16. PfA Methodology(2)• Conceptual extension from Logical Schema Partitioning andPhysical File Partitioning:4. Design your application to modularise its data access needsi. This is critical to segmenting your application to specific datafiles/filegroups.ii. This includes stored procedure and function internals, usetry/catch within sp’s for example to decide whether rollback orother functionality is needed when a filegroup is offline.iii. The key flows could be to either contain data access within the necessary filegroups, or determine whether a function can be used to catch that an object is in anoffline filegroup and skip it / do something else / roll back, or Catch “Msg 8653, Level 16, State 1, Line 3”* and do as per point above.5. Document application’s modules, with emphasis on criticalityand data dependencies.i. Allows stepped sequencing of apps by criticality during start upDesigning&DevelopingDatabasesforApplicationAvailability1726 July, 2010*[Tambs] While the database is in a state of partial availability, the filegroups thatremain online can support queries. However, queries that depend on data thatresides in filegroups that are offline return error 8653:Msg 8653, Level 16, State 1, Line 3The query processor is unable to produce a plan for the table or view X becausethe table resides in a filegroup which is not online.
  17. 17. 18Designing&DevelopingDatabasesforApplicationAvailability26 July, 2010AddingFiles and File Groups to a DatabaseUSE [master]GOCREATE DATABASE [PfA]ON PRIMARY(NAME = NPfA_Primary,FILENAME = NE:MSSQL10.SQL2008INST01MSSQLDataPfA_Primary.mdf,SIZE = 400MB),FILEGROUP [PfA_FG1](NAME = NPfA_F1_FG1,FILENAME = NE:MSSQL10.SQL2008INST01MSSQLDataPfA_F1_FG1.ndf,SIZE = 300MB),(NAME = NPfA_F2_FG1,FILENAME = NE:MSSQL10.SQL2008INST01MSSQLDataPfA_F2_FG1.ndf,SIZE = 300MB),FILEGROUP [PfA_FG2](NAME = NPfA_F1_FG2,FILENAME = NE:MSSQL10.SQL2008INST01MSSQLDataPfA_F1_FG2.ndf,SIZE = 300MB),FILEGROUP [PfA_FG3](NAME = NPfA_F1_FG3,FILENAME = NE:MSSQL10.SQL2008INST01MSSQLDataPfA_F1_FG3.ndf,SIZE = 300MB)LOG ON(NAME = NPfA_log,FILENAME = NF:MSSQL10.SQL2008INST01MSSQLLogsPfA_PfA_log.LDF,SIZE = 100MB)GOUSE [PfA];GOSELECT * FROM sys.database_files;SELECT * FROM sys.filegroups;
  18. 18. 19Designing&DevelopingDatabasesforApplicationAvailability26 July, 2010Create and Allocate Objectsto SchemasUSE [PfA];GOCREATE SCHEMA [Sales];GOCREATE SCHEMA [Production];GOCREATE SCHEMA [Person];GO-- moving tables from dbo schema to-- [Production] and [Sales] schemasALTER SCHEMA [Sales]TRANSFER [dbo].[Customer];-- Note that triggers move as well...GOALTER SCHEMA [Production]TRANSFER [dbo].[BillOfMaterials];GOALTER SCHEMA [Person]TRANSFER [dbo].[ContactInsert];GOALTER SCHEMA [Person]TRANSFER [dbo].[Contact];GO-- change stored proc definitions to focus on right schemasALTER PROC [dbo].[ContactInsert](@VbiContactID BIGINT,@VvcSalutation VARCHAR(4) = NULL,@VnvcForename NVARCHAR(100),@VnvcMiddlename NVARCHAR(100) = NULL,@VnvcSurname NVARCHAR(100),@VvcJobTitle NVARCHAR(100) = NULL,@VvcCompany NVARCHAR(100) = NULL,@VnvcAddressLine1 NVARCHAR(100),@VnvcAddressLine2 NVARCHAR(100) = NULL,@VnvcAddressLine3 NVARCHAR(100) = NULL,@VvcPostCode VARCHAR(12),@VnvcCity NVARCHAR(50),@VnvcState NVARCHAR(50) = NULL,@VinCountryID INT,@VvcEmailAddress VARCHAR(100),@VvcBusinessPhone VARCHAR(25),@VvcBusExt VARCHAR(10) = NULL,@VvcFacsimile VARCHAR(25) = NULL,@VvcMobilePhone VARCHAR(25) = NULL,@VvcPassword VARCHAR(20))ASINSERT INTO [Person].[Contact]([ContactID],[Salutation],[Forename],[Middlename],[Surname],[JobTitle],[Company],[AddressLine1],[AddressLine2],[AddressLine3],[PostCode],[City],[State],[CountryID],[EmailAddress],[BusinessPhone],[BusExt],[Facsimile],[MobilePhone],[Password])VALUES(@VbiContactID,@VvcSalutation,@VnvcForename,@VnvcMiddlename,@VnvcSurname,@VvcJobTitle,@VvcCompany,@VnvcAddressLine1,@VnvcAddressLine2,@VnvcAddressLine3,@VvcPostCode,@VnvcCity,@VnvcState,@VinCountryID,@VvcEmailAddress,@VvcBusinessPhone,@VvcBusExt,@VvcFacsimile,@VvcMobilePhone,@VvcPassword);GO* Use Synonyms to help the transition fromdbo native schema to proper schema usage
  19. 19. 20Designing&DevelopingDatabasesforApplicationAvailability26 July, 2010Allocatean Object to a FilegroupCREATE TABLE [Sales].[SalesInvoice]([SalesInvoiceID] INT IDENTITY(1,1) PRIMARY KEY CLUSTERED,[Amount] MONEY,[CustomerID] INT,[DateInvoiced] DATETIME,[DatePaid] DATETIME) ON [PfA_FG1];GO• Creating tables on a filegroup is easy as per the code sample.• Moving existing tables can be a little more complicated:• Use Create index (clustered) with Drop Existing• Rebuild indexes etc to move them• It may take many iterations to get the sequencing and code hereright, but its worth it in the long run.
  20. 20. 21Designing&DevelopingDatabasesforApplicationAvailability26 July, 2010Retry Logic (1)ADO.NET Code Example ( code example uses ADO.NET 2.0 (SqlClient). The following parameters are used in the example.strConn is the database connection string that includes the failover partner.strCmd is the command, such as Transact-SQL or a stored procedure, to execute.iRetryInterval is the number of seconds to wait (sleep) between retries.iMaxRetries is the number of times to retry the command before failing.Example connection string ( strConn )Data Source=SQLAINST1;Failover Partner=SQLBINST1;Initial Catalog=DBMTest;Integrated Security=Trueusing System.Data;using System.Data.SqlClient;using System.Threading;bool ExecuteSQLWithRetry_NoResults(string strConn,string strCmd,int iRetryInterval,int iMaxRetries){// Step 1: Enter the retry loop in case an error occursfor (int iRetryCount = 0; iRetryCount < iMaxRetries; iRetryCount++){try{// Step 2: Open the database connectionconn = new SqlConnection(strConn);conn.Open();// Step 3: Execute the commandif (null != conn && ConnectionState.Open == conn.State){cmd = new SqlCommand(strCmd, conn);cmd.ExecuteNonQuery();// Step 4: If there were no errors, clean-up & return successreturn true;}}catch (Exception ex){// Step 5: Catch Error on Open Connection or Execute Command// Error Handling to be added here: ex}• I expect that the code here will only needslight Adjustments for use with Clustering• Connection string would use virtual name• Timeout retry delay would cater for longerstartup/failover
  21. 21. 22Designing&DevelopingDatabasesforApplicationAvailability26 July, 2010Retry Logic (2)finally{ // Step 6: Clean up the Command and database Connection objectstry{// Clean Command objectif (null != cmd)cmd.Dispose();cmd = null;// Clean up Connection objectif (null != conn && ConnectionState.Closed != conn.State)conn.Close();conn = null;}catch (Exception ex){// Error handling to be added here}}// Step 7: If the maximum number of retries not exceededif (iRetryCount < iMaxRetries){// Step 7a: Sleep and continue (convert to milliseconds)Thread.Sleep(iRetryInterval * 1000);}} // end for loop: iRetryCount < iMaxRetries// Step 7b: If the max number of retries exceeded, return failurereturn false;}Continued…
  22. 22. 23Designing&DevelopingDatabasesforApplicationAvailability26 July, 2010Query Returning Files, Filegroups and SizesSELECTs.data_space_id as Id, as Filegroup_name, as Logical_name,physical_name,g.is_default AS [IsDefault],g.is_read_only AS [ReadOnly],CAST(ISNULL((SELECTSUM(gs.size)*CONVERT(float,8)FROMsys.database_files gsWHEREgs.data_space_id = g.data_space_id), 0) AS float) AS [Size]FROMsys.filegroups AS gINNER JOINsys.master_files AS sON (s.data_space_id = g.data_space_id)WHEREs.type = 0ANDs.database_id = db_id()ANDs.drop_lsn IS NULL; From “Partial Database Availability”(Danny Tambs, May 2007)
  23. 23. Links and Resources• AdventureWorks Database Diagram•• Whitepaper: “Partial Database Availability” (Danny Tambs, May 2007)•• Whitepaper: “Partitioned Table and Index Strategies Using SQL Server 2008” (RonTalmage, March 2009)• July, 2010
  24. 24. Questions?Designing&DevelopingDatabasesforApplicationAvailability2526 July, 2010