KoprowskiT - SQLBITS X - 2am a disaster just began
2 AMA DISASTER JUST BEGAN
ABOUT ME• Polish SQL Server User Group Leader• Microsoft Certified Trainer• MCP, MCSA, MLSS, MLSBS, MCTS, MCITP, MCT• SQL Server MVP• PASS SQL Azure Virtual Chapter Co-Founder• Blogger, Influencer, Technical Writer• Last 7 years in Data Center in Wrocław• Generally about 12 yeast in IT/banking area• GITCA Technical Lead & Vice-Chair EMEA Board• And As Freelancer: Trainer, Auditor, Consultant• Speaker at SQL Server Community Launch, Time for SharePoint, CodeCamps, SharePoint Community Launch, CISSP Day, InfoTRAMS• Deep Dives Co-Author: High availability of SQL Server in the context of Service Level Agreements (Chapter 18th)
2:00 AM … IN A DREAMS…• Your best time for dreaming … is the best time for Disaster• Your mobile phone ringing and ringing…• And Your husband / wife say…….
2:15 AM … IN A CAR• What’s happen with my server?• When I made last backup?• Where is my backup?• Have I ever tried restore? • If yes – I hope that all in a team (about team soon) remember about (me?) • If not – who can help me NOW?
2:40 AM … IN A SERVER ROOM• $#$$@$^^#^&^@!#• Is Windows Server alive? • YES (thanks all saints) • or NOT (damm) • who is responsible for it?• Is my SQL Server alive? • YES (why phones ringing) • or NOT (…) • Why I’m responsible for it?
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPARED
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPAREDBackups• about type of backup ( simple rm / full rm)• about place for stored backup data• about backup window• about procedure used for backup• about backup tools• about backup of „backup logs”• About estimated time for executint backup• About REAL TIME of executing backup
BACKUP > EXTRACT FROM SOP*In the request, backup, should include the following information:• Information about the operating system and application versionfor online backup and installed updates for these components• a file backup policy, in particular:a number of versions of a file storedThe storage time of the next version of the fileThe frequency of execution of such incremental backups with theproposal of their implementation• Online Backup PolicyThe storage time of a full backupwith storage time such an incremental backupthe time of transaction log filesThe frequency of execution of a full backup with the proposal deadline forits implementationThe frequency of execution of the transaction log backup• Information about trees directories / files that should beomittedor included during backup tasks (include / exclude list)• Number and type / model of physical processors, operating int he node-connected• Does the node will use the connection to the SAN to implement backup
BACKUP (REGISTRY) > EXTRACT FROM SOP*This register contains information about a backup plan implemented. Itconsists of the following fields:Backup file space: number of versions of a file stored in a backup number of days that are kept more versions of a file number of versions of a file stored in the backup system after itsremoval from the client device the number of days that will store the latest version of thedeleted file from the client device the number of days that will be stored in the archiveOnline Backup: the number of backups stored in full backup number of incremental backups / difffull stored in the backup Frequency of transaction log backups stored in the backup (fordatabases) the number of days the backups to be stored on-line backupsystemThe list of nodes defined in the system backup: Domain | Node name | IP address of the nodeThe list of defined backup tasks (called scheduli) name of the task (schedulea) | execution time a period of at which the task is repeated
BEST PRACTICES BY BRENT OZARSQL Server Backup Best Practices | Written on October 17, 2007 by Brent Ozar in SQLServer >> http://www.brentozar.com/archive/2007/10/backing-up-sql-server-my-own-mediocre-practices/I’ve been backing up SQL Servers for almost a decade now, and it’s time to share thelessons I’ve learned. All of this is my own opinion – your mileage may vary – but I’lltry to explain the reasoning behind the choices I make. I won’t address log shippingor snapshots this time around.• Never back up databases to local disk.• Back up databases to a fileshare, then back the share up to tape.• Cost justify the network share with lower licensing costs & simpler backups.• Back up to a different SAN if possible.• My sweet spot for the backup array is raid 10 SATA.• Backup agents like NetBackup and Backup Exec mean giving up scheduling control.• Do regular fire drill rebuilds and restores.• Build a standalone restore testbed.• Keep management informed on restore time estimates.• Trust no one.
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPAREDRestore• about type of backup ( simple rm / full rm)• about place for stored backup data• about the procedures of recovery• about estimated time for recovery• about REAL TIME for recovery• about tools for recovery• about Corporate Backup Manager• about password for access to library
RESTORE > EXTRACT FROM SOP*Register for Recovery/Restore/ReplacementTestsThis register contains information about the tests and replacement ofpart or all of the environment.It consists of the following fields: the date of commencement and completion of the recovery test client for which the test was performed recovery test servers involved in testing and replacement extent of testing and replacement the person / persons performing the recovery test the person on the client side accepts the correctness of the recoverytest subsequent to the recovery test
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPAREDProcedures• It is not about stored procedures!!!• It’s about storing procedures with answers for the following: • One piece of paper • How to start restore • Who can help • How to processing a restore • When we can finish• It MUST be simple
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPAREDRoles• Database Administrator• Windows Administrator• Backup Administrator• Network Administrator• Customer Key Account• Manager of division• Data Center Manager• Nightshift Operator• Customer Administrator!!
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPAREDPSO > USO > SLA• PSO Planned System Outages – Planned System Unavailability • Minimum planned unavailability, due to the need to carry out modernization work, installing patches, replacement / extension of hardware, • Agreed/accepted by/with the client and not affecting the provisions of the HA, and SLA, until• ...USO Unplaned System Outages – Unplanned System Unavailability • an error that prevents a partial or total work environment in a tangible, measurable customer • resulting in high costs if you need repairs, as well as penalty payments for non- SLA
THE MAGIC NINES… Downtime per Downtime per Downtime per Availability % year month* week90% 36.5 days 72 hours 16.8 hours95% 18.25 days 36 hours 8.4 hours98% 7.30 days 14.4 hours 3.36 hours99% 3.65 days 7.20 hours 1.68 hours99.5% 1.83 days 3.60 hours 50.4 min99.8% 17.52 hours 86.23 min 20.16 min99.9% ("three nines") 8.76 hours 43.2 min 10.1 min99.95% 4.38 hours 21.56 min 5.04 min99.99% ("four nines") 52.6 min 4.32 min 1.01 min99.999% ("five nines") 5.26 min 25.9 s 6.05 s99.9999% ("six nines") 31.5 s 2.59 s 0.605 s
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPAREDEnvelopeWith ACTUAL!!! User names and passwords for:• Windows Server Administrator• SQL Server Administrator• SQL Server Agent• SQL Server Services (if You didn’t use default)• SQL Server Applications Services• Backup accountss
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPAREDHardwareSome of the hard stuff for replacement:• Server• Motherboard• Memory (RAM)• Processor (CPU)• Network Adapter (LAN/NIC)• Fibre Channel Adapter• Hard Disk (IDE/SATA/SAS/SSD…)• RAID Controller
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPAREDSoftware• Windows • 2000/2003/2003R2/2008/2008R2 • SP 1,2,3,4 +CU 1, 2,3, … • Standard, Enterprise, Data Center • x32, x64, ia64• SQL Server • 6.5, 7.0, 2000, 2005, 2008, 2008R2, 2012 (haha) • SP 1,2,3,4 +CU, 1,2,3,4,5,6,7,8,9…. • Drivers (servers, lan card, video card)• AGENT ORANGE
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPAREDKeysSome keys which You need…• Serial keys• Rack keys• Server keys• Storage keys• Knife• Lighter• Phone
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPAREDENCRYPTIONIf You use encryption (such a TDE)• TDE • Create encryption key • Export encryption key • Backup encryption key• CA • Remember about expiration date• BitLocker
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPAREDTEAMYou can work with disaster as:Team MemberTeam LeaderLast Samurai
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPAREDMANAGERShmm
BEST PRACTICE FOR SURVIVING DISASTERITS ONLY ONE: BE PREPARED• Backups (and know-how about stored place, and restoring way)• Procedures (the short is better | one page is the best)• Roles (who can help, who is necessary for access)• SLA (90? 95? 99,99? in minutes, hours or days you have to recover)• Envelope (with user names and passwords for all important accounts)• Hardware (server, motherboard, CPU, RAM, LAN, HDD, SDD, USB)• Software (Windows+SP+CU, SQL+SP+CU, drivers, AGENT ORANGE)• Keys (serial numbers, physical keys, knife)• ENCRYPTION (arrghhhhh!!! Certificates, keys, internal/external)• TEAM (Team, leader, separate…)• MANAGERS (hmmm)