VLDB Administration Strategies

593 views

Published on

Presentation done by SQL Port event.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
593
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
15
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • Track = Group of Sectors and CLustersCluster = File allocation Unit – Um Cluster tem N Sectors – Para um allocation unit de 64K -> 512 b (sector size) * 128 = 64K!!Sector = Smallest accessible unit in a physical disk (usually 512 bytes)Stripe Unit Size = Define o tamanhoqueos dados serãodistribuidos entre os discos de um grupo RAID-0, RAID-5, RAID-10
  • Track = Group of Sectors and CLustersCluster = File allocation Unit – Um Cluster tem N Sectors – Para um allocation unit de 64K -> 512 b (sector size) * 128 = 64K!!Sector = Smallest accessible unit in a physical disk (usually 512 bytes)Stripe Unit Size = Define o tamanhoqueos dados serãodistribuidos entre os discos de um grupo RAID-0, RAID-5, RAID-10
  • Track = Group of Sectors and CLustersCluster = File allocation Unit – Um Cluster tem N Sectors – Para um allocation unit de 64K -> 512 b (sector size) * 128 = 64K!!Sector = Smallest accessible unit in a physical disk (usually 512 bytes)Stripe Unit Size = Define o tamanhoqueos dados serãodistribuidos entre os discos de um grupo RAID-0, RAID-5, RAID-10
  • Track = Group of Sectors and CLustersCluster = File allocation Unit – Um Cluster tem N Sectors – Para um allocation unit de 64K -> 512 b (sector size) * 128 = 64K!!Sector = Smallest accessible unit in a physical disk (usually 512 bytes)Stripe Unit Size = Define o tamanhoqueos dados serãodistribuidos entre os discos de um grupo RAID-0, RAID-5, RAID-10
  • Track = Group of Sectors and CLustersCluster = File allocation Unit – Um Cluster tem N Sectors – Para um allocation unit de 64K -> 512 b (sector size) * 128 = 64K!!Sector = Smallest accessible unit in a physical disk (usually 512 bytes)Stripe Unit Size = Define o tamanhoqueos dados serãodistribuidos entre os discos de um grupo RAID-0, RAID-5, RAID-10
  • Track = Group of Sectors and CLustersCluster = File allocation Unit – Um Cluster tem N Sectors – Para um allocation unit de 64K -> 512 b (sector size) * 128 = 64K!!Sector = Smallest accessible unit in a physical disk (usually 512 bytes)Stripe Unit Size = Define o tamanhoqueos dados serãodistribuidos entre os discos de um grupo RAID-0, RAID-5, RAID-10
  • Track = Group of Sectors and CLustersCluster = File allocation Unit – Um Cluster tem N Sectors – Para um allocation unit de 64K -> 512 b (sector size) * 128 = 64K!!Sector = Smallest accessible unit in a physical disk (usually 512 bytes)Stripe Unit Size = Define o tamanhoqueos dados serãodistribuidos entre os discos de um grupo RAID-0, RAID-5, RAID-10
  • Track = Group of Sectors and CLustersCluster = File allocation Unit – Um Cluster tem N Sectors – Para um allocation unit de 64K -> 512 b (sector size) * 128 = 64K!!Sector = Smallest accessible unit in a physical disk (usually 512 bytes)Stripe Unit Size = Define o tamanhoqueos dados serãodistribuidos entre os discos de um grupo RAID-0, RAID-5, RAID-10
  • Track = Group of Sectors and CLustersCluster = File allocation Unit – Um Cluster tem N Sectors – Para um allocation unit de 64K -> 512 b (sector size) * 128 = 64K!!Sector = Smallest accessible unit in a physical disk (usually 512 bytes)Stripe Unit Size = Define o tamanhoqueos dados serãodistribuidos entre os discos de um grupo RAID-0, RAID-5, RAID-10
  • Track = Group of Sectors and CLustersCluster = File allocation Unit – Um Cluster tem N Sectors – Para um allocation unit de 64K -> 512 b (sector size) * 128 = 64K!!Sector = Smallest accessible unit in a physical disk (usually 512 bytes)Stripe Unit Size = Define o tamanhoqueos dados serãodistribuidos entre os discos de um grupo RAID-0, RAID-5, RAID-10
  • Track = Group of Sectors and CLustersCluster = File allocation Unit – Um Cluster tem N Sectors – Para um allocation unit de 64K -> 512 b (sector size) * 128 = 64K!!Sector = Smallest accessible unit in a physical disk (usually 512 bytes)Stripe Unit Size = Define o tamanhoqueos dados serãodistribuidos entre os discos de um grupo RAID-0, RAID-5, RAID-10
  • Track = Group of Sectors and CLustersCluster = File allocation Unit – Um Cluster tem N Sectors – Para um allocation unit de 64K -> 512 b (sector size) * 128 = 64K!!Sector = Smallest accessible unit in a physical disk (usually 512 bytes)Stripe Unit Size = Define o tamanhoqueos dados serãodistribuidos entre os discos de um grupo RAID-0, RAID-5, RAID-10
  • It depends:HBAs or FusionIO storage, both of which require a lot of free memory for the drivers.Concurrent applications in the server.Number of instance instance or SQL Services
  • 32-bit applications are natively restricted to a 2 GB VAS/3GB allows a 32-bit process to increase its VAS to 3 GBTo address more than 4 GB of RAM on 32-bit Windows, the OS needs to have the /PAE switch added to the boot.ini file
  • chunks less than 64MB and up to 64MB = 4 VLFschunks larger than 64MB and up to 1GB = 8 VLFschunks larger than 1GB = 16 VLFsHuge t-logs cause huge VLFsHuge VLFs are hard to clear (performance issue)SQL Server can only clear (backup) inactive VLFs, so a huge VLF can take time to be free and will take time to be backed up.
  • chunks less than 64MB and up to 64MB = 4 VLFschunks larger than 64MB and up to 1GB = 8 VLFschunks larger than 1GB = 16 VLFsHuge t-logs cause huge VLFsHuge VLFs are hard to clear (performance issue)SQL Server can only clear (backup) inactive VLFs, so a huge VLF can take time to be free and will take time to be backed up.
  • chunks less than 64MB and up to 64MB = 4 VLFschunks larger than 64MB and up to 1GB = 8 VLFschunks larger than 1GB = 16 VLFsHuge t-logs cause huge VLFsHuge VLFs are hard to clear (performance issue)SQL Server can only clear (backup) inactive VLFs, so a huge VLF can take time to be free and will take time to be backed up.
  • VLDB Administration Strategies

    1. 1. @murilocmiranda http://www.sql.pt/ murilo.miranda@gmail.com
    2. 2. AGENDA
    3. 3. 1. 2. 3. 4. 5. 6. What is a VLDB? Typical Troubles OS Config Instance Config DB Config Maintenance AGENDA
    4. 4. VLDB??
    5. 5. There’s no official definition. VLDB??
    6. 6. There’s no official definition. Typically occupying TB range. VLDB??
    7. 7. There’s no official definition. Typically occupying TB range. Billions of rows. VLDB??
    8. 8. There’s no official definition. Typically occupying TB range. Billions of rows. Typically: OLAP or OLTP with large amount of users. VLDB??
    9. 9. Wikipedia… A very large database, or VLDB, is a database that contains an extremely high number of tuples (database rows), or occupies an extremely large physical filesystem storage space. The most common definition of VLDB is a database that occupies more than 1 terabyte or contains several billion rows, although naturally this definition changes over time. VLDB??
    10. 10. SQL VS. VLDB
    11. 11. Maximum database size SQL VS. VLDB
    12. 12. Maximum database size 524,272 TB SQL VS. VLDB
    13. 13. Maximum data file size Maximum log file size 16 TB 2 TB A limit of 32.767 files which can be distributed between 32.767 filegroups. SQL VS. VLDB
    14. 14. TYPICAL TROUBLES
    15. 15. Maintenance TYPICAL TROUBLES
    16. 16. Backups Maintenance TYPICAL TROUBLES
    17. 17. Backups Maintenance Indexes TYPICAL TROUBLES
    18. 18. Backups Maintenance Indexes Statistics TYPICAL TROUBLES
    19. 19. Backups Maintenance Indexes Statistics Disaster Recovery TYPICAL TROUBLES
    20. 20. Backups Maintenance Indexes Performanc e Statistics Disaster Recovery TYPICAL TROUBLES
    21. 21. OS CONFIG
    22. 22. Perform Volume Maintenance OS CONFIG
    23. 23. Turning on Instant Initialization to speed up data file growth and restores. OS CONFIG
    24. 24. Storage Layout OS CONFIG
    25. 25. Plan an efficient storage layout. OS CONFIG
    26. 26. Plan an efficient storage layout. Normally, the more spread, the more effective. OS CONFIG
    27. 27. Plan an efficient storage layout. Normally, the more spread, the more effective. Suggestion : SQL BIN SQL DATA SQL IDX SQL LOGS SQL TMP OS CONFIG
    28. 28. Mountpoint s OS CONFIG
    29. 29. Mountpoints could be a good strategy. OS CONFIG
    30. 30. Mountpoints could be a good strategy. Mountpoints are persistent directories that point to disk volumes. OS CONFIG
    31. 31. Pros: • • • • Scalable. Save drive letters (limited to 26). Easy to add. No need to restart SQL Server. OS CONFIG
    32. 32. Cons: • Looks like a simple folder. • Need a different approach to monitor. OS CONFIG
    33. 33. So, if you don’t know the server…. OS CONFIG
    34. 34. Partition Alignment OS CONFIG
    35. 35. Setting the partition offset properly can improve up to 30% the performance. OS CONFIG
    36. 36. Setting the partition offset properly can improve up to 30% the performance. Partition alignment increases throughput (bytes/sec) and reduce disk queues. OS CONFIG
    37. 37. Setting the partition offset properly can improve up to 30% the performance. Partition alignment increases throughput (bytes/sec) and reduce disk queues. A partition that is track misaligned will occasionally cause 2 I/O operations instead of one. OS CONFIG
    38. 38. Unless performed at the time of partition creation, the default alignment offset (31,5 Kb) will result in unaligned partitions on versions of Windows up to and including Windows Server 2003. OS CONFIG
    39. 39. This offset is associated with hidden sectors, which basically store partition information. OS CONFIG
    40. 40. This offset is associated with hidden sectors, which basically store partition information. Considering that: - Each disk sector has 512 bytes. - Win. 2003 has 63 hidden sectors. OS CONFIG
    41. 41. This offset is associated with hidden sectors, which basically store partition information. Considering that: - Each disk sector has 512 bytes. - Win. 2003 has 63 hidden sectors. 512 * 63 = 31,5 Kb OS CONFIG
    42. 42. Example: Stripe Unit Size: Allocation Unit Size: Optimal values 64Kb* 64Kb OS CONFIG * Defined by storage team.
    43. 43. Example: Stripe Unit Size: Allocation Unit Size: Optimal values 64Kb* 64Kb Data (Alloc. Unit Size) OS CONFIG Stripe Size * Defined by storage team.
    44. 44. Optimal solution: Data (Alloc. Unit Size) Stripe Size OS CONFIG
    45. 45. Best Practice: - Set an offset of 1024 Kb. - This value works for mostly disks out there. - Allocation Unit Size = Stripe Unit Size. The rule: Offset / Allocation unit = INTEGER Eg: 1024/64=16 OS CONFIG
    46. 46. WARNIG Some I/O subsystem vendors intercepting what Windows is trying to do and are still creating partitions with the incorrect offset – Even for Windows 2008+. ALWAYS check!
    47. 47. Anti-Virus servers… is really a need? OS CONFIG in
    48. 48. • • • • Cost money to license. Maintenance costs. Can cause problems in Prod. Can’t protect to zero-day exploits. OS CONFIG
    49. 49. What can we do instead? OS CONFIG
    50. 50. • • • • Keep the servers patched. Configure the firewall properly. Restrict server’s access. You can install AV… in workstations! OS CONFIG
    51. 51. What’s the big problem for SQL Server? OS CONFIG
    52. 52. • One more app fighting for resources. • SQL Server files can be locked. OS CONFIG
    53. 53. How can AV and SQL Server live together? OS CONFIG
    54. 54. Add exceptions! OS CONFIG
    55. 55. Basically the AV should ignore: • • • • • • • SQL Server data and log files (.mdf, .ndf and .ldf). Backup files (.bak and .trn). Full-text Catalog files. Trace files (.trc). ERRORLOG files. SQL Server binaries folder. Filestream folder. OS CONFIG More on: http://support.microsoft.com/kb/309422
    56. 56. INSTANCE CONFIG
    57. 57. Memory INSTANCE CONFIG
    58. 58. Memory This is a very open subject. INSTANCE CONFIG
    59. 59. Memory This is a very open subject. There are lots of discussions about that… INSTANCE CONFIG
    60. 60. Memory This is a very open subject. There are lots of discussions about that… There’s no perfect formula, because the correct awnser is…. INSTANCE CONFIG
    61. 61. Memory This is a very open subject. There are lots of discussions about that… There’s no perfect formula, because the correct answer is…. … it depends !! INSTANCE CONFIG
    62. 62. Memory An efficient general rule… Baseline: 1 GB for the OS Up to 16 GB available • 1 GB for each 4 GB More than 16 GB • 1 GB for every 8 GB INSTANCE CONFIG
    63. 63. Memory This is for 64 bit servers… For 32 bit, here is a good article to follow: http://www.eraofdata.com/understanding-and-configuring-sql-servers-memory-settings/ INSTANCE CONFIG
    64. 64. TempDB INSTANCE CONFIG
    65. 65. TempDB Two common behaviors: INSTANCE CONFIG
    66. 66. TempDB Two common behaviors: • Ignore. • Overvalue. INSTANCE CONFIG
    67. 67. TempDB As per Brent Ozar: “TempDb is the SQL’s public toilet” INSTANCE CONFIG
    68. 68. TempDB And this is true! INSTANCE CONFIG
    69. 69. TempDB INSTANCE CONFIG
    70. 70. TempDB There’s a myth: • tempdb should always have one data file per processor core. INSTANCE CONFIG
    71. 71. TempDB There’s a myth: • tempdb should always have one data file per processor core. Again…. INSTANCE CONFIG
    72. 72. TempDB There’s a myth: • tempdb should always have one data file per processor core. Again…. It depends! INSTANCE CONFIG
    73. 73. TempDB Execute large operations, like a sort or store a huge temporary table, may be slowed down because of the round-robin operation. The more files, the more costly. INSTANCE CONFIG
    74. 74. TempDB Common wait types on TempDB: • PAGELATCH_*: Contention for In-memory allocation bitmaps. • PAGEIOLATCH_*: Contention at the I/O subsystem level. INSTANCE CONFIG
    75. 75. TempDB How many tempdb data files should we have? INSTANCE CONFIG
    76. 76. TempDB How many tempdb data files should we have? A recommended approach is: • Up to 8 cores: Number of files = Number of cores. • More than 8 cores: 1. Add 8 files. 2. Monitor PAGELATCH_*. 3. Add 4 more files at a time, if necessary. INSTANCE CONFIG
    77. 77. TempDB Other TempDB best practices: • Isolate the TempDB in a different storage system. • Depending of the load, you might need to separate LDF and M(N)DF. • Use a fast drive (SSD :). • Set an initial size, equally to all the files. • Set the auto-growth accordingly. • If you have a heavy operation using constantly the TempDB, consider create a staging table into your own database. INSTANCE CONFIG
    78. 78. TempDB From SQL Server 2012, local disk TempDB in SQL Server cluster. INSTANCE CONFIG
    79. 79. TempDB From SQL Server 2012, local disk TempDB in SQL Server cluster. • More flexibility. • Use PCIe bus instead of HBA, and have more throughput. • Data and Log are in SAN, TempDB locally: Avoid congestion or contention on a shared storage network or array. INSTANCE CONFIG
    80. 80. DB CONFIG
    81. 81. • Don’t rely on auto-grow. • You can manage file growth and control the free disk space and avoids performance problems. DB CONFIG
    82. 82. • Don’t rely on auto-grow. • You can manage file growth and control the free disk space and avoids performance problems. • Have page checksums turned on. • To detect damaged pages. DB CONFIG
    83. 83. • Don’t rely on auto-grow. • You can manage file growth and control the free disk space and avoids performance problems. • Have page checksums turned on. • To detect damaged pages. • Make sure auto-stats update is turned on. • For OLTP consider turning auto-stats update off only for heavily updated tables, and schedule a job that periodically updates the statistics for those tables. DB CONFIG
    84. 84. DB CONFIG
    85. 85. • Make sure you’re managing the transaction log correctly: • Full recovery requires log backups. • No advantage in have multiple log files. • Control the file growth or this could cause VLF fragmentation. • Performance issues. • Slow backup time. • Don’t set the log file growth size to a multiple of 4 in older SQL Server versions. • http://connect.microsoft.com/SQLServer/feedback/details/481594/log-growth-not-workingproperly-with-specific-growth-sizes-vlfs-also-not-created-appropriately DB CONFIG
    86. 86. MAINTENACE
    87. 87. Few questions… MAINTENANCE
    88. 88. How to meet your SLAs dealing with a TB database? Is data-loss acceptable? What about the recovery time? Are you able to UPDATE STATS, do INDEX MAINTENANCE and run a INTEGRITY CHECK in time and WITHOUT PROBLEMS? MAINTENANCE
    89. 89. DISASTER RECOVERY MAINTENANCE
    90. 90. First of all, think in a Disaster Recovery plan! SQL Server is not Oracle, we have “free” included options: • Log Shipping (HA and DR) • Database Mirroring (HA and DR) • DB Snapshot advantage • Replication (HA, DR and LB) • AlwaysOn (HA, DR and LB) • We can still be safe with a storage level replication. MAINTENANCE
    91. 91. Partition Compress Clean MAINTENANCE
    92. 92. Partition, Compress and Clean Using the partitioning feature you can devise the maintenance. MAINTENANCE
    93. 93. Partition, Compress and Clean Using the partitioning feature you can devise the maintenance. • You can use the DBCC CHECKFILEGROUP command. • DBCC CHECKFILEGROUP and DBCC CHECKDB are. The main difference is that DBCC CHECKFILEGROUP is limited to the single specified filegroup and required tables. MAINTENANCE
    94. 94. Partition, Compress and Clean Using the partitioning feature you can devise the maintenance. • Devising a filegroup architecture allows piecemeal restores with low TTR • Online piecemeal restore: • • After the PRIMARY FG restore the DB can be online. The tables will come available while each FG is restored. • Design the database accordingly: • • Keep the necessary into the PRIMARY FG. • Configuration tables, indispensable data, etc… Think in the consistency: keep related tables in the same FG. MAINTENANCE
    95. 95. Partition, Compress and Clean Compress backups Vs. Compress Data • Backup compression: • More CPU usage to backup/restore (avg ~20%). • Less time to backup/restore (avg ~40%). • Good compression ratio. • SELECT backup_size/compressed_backup_size FROM msdb..backupset; • A backup set will not be able to contain both compressed and uncompressed backups. • No advantage with TDE enabled. MAINTENANCE
    96. 96. Partition, Compress and Clean Compress backups Vs. Compress Data • Data compression (ROW and PAGE): • TDE and Data Compression play together! • Backup and Data Compression can coexist! MAINTENANCE
    97. 97. Partition, Compress and Clean Purge and Archive the data • Purging data: • If data is needed no more… • Save storage. • Faster backups. • Improves the performance. MAINTENANCE
    98. 98. Partition, Compress and Clean Purge and Archive the data • Archiving data: • If data is still needed… • Isolate in a different FG. • Set as Read-Only: Avoids locking. • For faster scans: 100% fill factor. • Update statistics with FULLSCAN. • You can adapt the backup strategy. • You can adapt the backup strategy using Partial Backups. MAINTENANCE • This allows you to exclude read-only filegroups.
    99. 99. More about DBCC CHECKDB • CHECKDB takes time and uses resources. • Run a DBCC CHECKDB using the WITH PHYSICAL_ONLY option. • • Limits the checking to the integrity of the physical structure of the page and record headers and the allocation consistency of the database. Faster, but a full CHECKDB is required periodically. MAINTENANCE
    100. 100. More about DBCC CHECKDB • We can divide up the consistency checking over several days, Paul Randal’s prescription is: • Divide tables in two buckets (bigger ones and the rest) • On Sunday: • Run a DBCC CHECKALLOC • Run a DBCC CHECKCATALOG • Run a DBCC CHECKTABLE on each table in the first bucket • On Monday, Tuesday, Wednesday: • Run a DBCC CHECKTABLE on each table in the 2nd, 3rd, 4th buckets, respectively • On Thursday: • Run a DBCC CHECKALLOC • Run a DBCC CHECKTABLE on each table in the 5th bucket • On Friday and Saturday: • Run a DBCC CHECKTABLE on each table in the 6th and 7th buckets, respectively MAINTENANCE More on: http://www.sqlskills.com/blogs/paul/checkdb-from-every-angle-consistency-checking-options-f
    101. 101. More about BACKUPS • Besides doing PARTIAL BACKUPS we have more options… • A MULTISTREAM BACKUP is an option to run faster: File 1 File 2 F: File 3 DB E: G: MAINTENANCE
    102. 102. More about BACKUPS • To make sure it will be well stored, we can use a MIRROR. File 1 File 1 File 2 File 3 DB E: File 2 F: File 3 G: MAINTENANCE
    103. 103. More about BACKUPS • If storing to the network: • Use a separate network card to avoid network congestion. • Don’t forget about T-LOG backups! • Create a good backup strategy. • Verify the backups periodically. MAINTENANCE
    104. 104. INDEXES MAINTENANCE • Only rebuild/defrag indexes that are really fragmented (avoid unnecessary work in short maintenance windows) • If you defrag instead of rebuild, make sure you manually update stats. • Be wary of doing large index maintenance jobs if you use log shipping or DBM • They contribute to large log backups • Index rebuilds are always full-logged when DBM is present MAINTENANCE
    105. 105. QUESTIONS?
    106. 106. OBRIGADO! @murilocmiranda http://www.sql.pt/ murilo.miranda@gmail.com

    ×