Large Scale SQL Considerations for SharePoint Deployments

6,169 views

Published on

Large Scale SQL Considerations for SharePoint: Considerations for performance, scale, storage, and high availability

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,169
On SlideShare
0
From Embeds
0
Number of Embeds
238
Actions
Shares
0
Downloads
245
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Large Scale SQL Considerations for SharePoint Deployments

  1. 1. Joel Oleson Sr. Product Architect Quest Software http://www.sharepointjoel.com @joeloleson Contributions: Mike Watson, Todd Klindt
  2. 2. Audience Poll <ul><li>New to SharePoint? </li></ul><ul><li>SQL Admins? </li></ul><ul><li>SharePoint Admins? </li></ul><ul><li>Large-scale Implementation (+1 TB) experience? </li></ul><ul><li>How many SQL Admins are freaking out because of the number of SharePoint databases? </li></ul>
  3. 3. Session Objectives And Takeaways <ul><li>Session Objective(s): </li></ul><ul><ul><li>Understand the SQL and storage factors that affect a large scale SharePoint deployment. </li></ul></ul><ul><ul><li>SharePoint SQL and storage best practices. </li></ul></ul><ul><li>Takeaway: </li></ul><ul><ul><li>Proper SQL and Storage design is critical to overall SharePoint health! </li></ul></ul>
  4. 4. SharePoint Databases Overview
  5. 5. SharePoint Containment Hierarchy
  6. 6. Understanding SharePoint Databases
  7. 7. Understanding Configuration DB
  8. 8. Understanding Content DB
  9. 9. Understanding SSP DB - Search
  10. 10. Understanding SSP DB
  11. 12. Why is SQL that important? SQL Health = SharePoint Health! <ul><li>Sub-optimal SQL perf will radiate to other components in the farm. </li></ul><ul><ul><li>Slow response from SQL Server will result in queued App requests. </li></ul></ul><ul><ul><li>As the app slows down, so does SQL. </li></ul></ul>
  12. 13. Database Disk I/O Demand Most Demand Medium Demand Low Demand * Except during backup and Indexing + Except during Profile Import Temp Master Model Tlogs Search Config +SSP *Content..
  13. 14. Top Performance Killers <ul><li>Indexing/Crawling </li></ul><ul><li>Backup (SQL & Tape) </li></ul><ul><li>Profile Import </li></ul><ul><li>Misc Timer Jobs – User Sync for large #s of Users </li></ul><ul><li>Poor Storage Configuration </li></ul><ul><li>STSADM Backup/Restore </li></ul><ul><li>Large List Operations </li></ul><ul><li>Heavy User Operation List Import/Write </li></ul><ul><li>Network </li></ul><ul><li>Inefficient Queries </li></ul>
  14. 16. Scaling SQL 2.5TB SCALE OUT  2.5TB 2.5TB
  15. 17. Scalling SQL - Out <ul><li>More SQL servers = More flexibility </li></ul><ul><li>There aren’t really any physical barriers </li></ul><ul><li>SharePoint won’t prevent you from placing 100 databases on 100 different SQL instances </li></ul><ul><li>The real barriers are manageability and cost. </li></ul><ul><ul><li>More servers = more money </li></ul></ul><ul><ul><li>More servers = more management </li></ul></ul><ul><ul><li>$$ + > management = $$$$ </li></ul></ul>
  16. 18. Scaling SQL 2.5TB SCALE UP  2.5TB 2.5TB
  17. 19. Scaling SQL - Up <ul><ul><li>Design is Paramount! </li></ul></ul><ul><ul><li>Consider the following: </li></ul></ul><ul><ul><ul><li>Overall SQL Throughput (transactions/sec) </li></ul></ul></ul><ul><ul><ul><li>Disk throughput (IOPS) </li></ul></ul></ul><ul><ul><ul><li>Network throughput (MB/sec) </li></ul></ul></ul><ul><ul><ul><li>Disk backup throughput (MB/sec) </li></ul></ul></ul><ul><ul><ul><li>Network based backup throughput (MB/sec) </li></ul></ul></ul><ul><ul><ul><li>Length of maintenance windows (hours -> minutes) </li></ul></ul></ul><ul><ul><ul><li>SharePoint upgrade throughput </li></ul></ul></ul>
  18. 20. SQL: Scale Out VS. Scale Up Scale Out Scale Up Advantages Better Performance Easier to Manage Better Flexibility Cheaper Disadvantages More Expensive System Design is Critical Harder to Manage Single Point of Failure
  19. 21. Walkthrough: Scale Up VS. Out How to design a 5TB SharePoint SQL Deployment 1TB 1TB 1TB 1TB 1TB 1TB 1TB 1TB 1TB 1TB
  20. 22. Consider the Organization <ul><li>Will the SharePoint SQL Servers be self managed? </li></ul><ul><li>What experience does the team managing SQL have? </li></ul><ul><li>Do they have: </li></ul><ul><ul><li>Monitoring? </li></ul></ul><ul><ul><li>Standard Maintenance Procedures? </li></ul></ul><ul><ul><li>Standard Maintenance Windows? </li></ul></ul><ul><ul><li>Standard SQL Builds? </li></ul></ul><ul><ul><li>What are the break/fix and standard SLA’s? </li></ul></ul>
  21. 23. Scaling SQL – The Bottom Line <ul><li>Don’t scale SQL instances beyond comfort zones! </li></ul><ul><li>Do measure system throughput – Know All of your bottlenecks! </li></ul><ul><li>Scaling out is more flexible but scaling up is more cost effective. Find a balance between scaling up and out and stick to it. (1-5TB per instance for example) </li></ul>
  22. 25. Highly Available Deployment? Redundant Switches Redundant Web/Application Servers Active/Passive SQL w/ Redundant HBA’s Redundant SAN Fabric RAID 1 Storage Redundant Power Supplies
  23. 26. Mirroring Within a Farm <ul><li>SQL High Avail or High Protection (sync) mirroring replaces or augments clustering as the SQL HA solution. </li></ul><ul><li>Farm components can span closely located datacenters* </li></ul><ul><ul><li>Must have LAN like connectivity (1Gbps) </li></ul></ul><ul><ul><li>Must have less than 1ms in latency (2ms RTT) </li></ul></ul><ul><li>Can be Active/Active or Active/Passive </li></ul><ul><li>Use DNS or Load Balancing to direct traffic between frontends. </li></ul>
  24. 27. Mirroring Within Farm
  25. 28. High Availability Between Farms <ul><li>Can use a variety of methods to ship content between farms/data centers </li></ul><ul><ul><li>Log shipping </li></ul></ul><ul><ul><li>Mirroring </li></ul></ul><ul><ul><li>Storage replication </li></ul></ul><ul><li>Longer distances supported* </li></ul><ul><ul><li>The greater the latency the harder it is to replicate content. </li></ul></ul><ul><li>No way to keep configuration or search in sync. </li></ul>
  26. 29. High Availability Between Farms
  27. 30. The Two Basic HA/DR Scenarios <ul><li>Mirroring Within Farm </li></ul><ul><li>Pros: </li></ul><ul><ul><li>Great combo HA/DR solution </li></ul></ul><ul><ul><li>Cheaper to implement </li></ul></ul><ul><ul><li>Easier to manage </li></ul></ul><ul><li>Cons: </li></ul><ul><ul><li>Requires closely located datacenters </li></ul></ul><ul><ul><li>Requires excellent network conditions </li></ul></ul><ul><ul><li>Not flexible </li></ul></ul><ul><ul><li>Content corruption is replicated immediately. </li></ul></ul><ul><li>Mirroring/Log ship Between Farms </li></ul><ul><li>Pros: </li></ul><ul><ul><li>Allows long distance separation </li></ul></ul><ul><ul><li>Can protect against logical corruption </li></ul></ul><ul><ul><li>Very flexible! </li></ul></ul><ul><li>Cons: </li></ul><ul><ul><li>More expensive </li></ul></ul><ul><ul><li>Harder to setup and manage </li></ul></ul><ul><ul><li>Failover is a big decision </li></ul></ul>
  28. 31. Combining Solutions
  29. 32. SQL 2008 - Do you have Enterprise?
  30. 34. Content DB Size Limitation 100GB? <ul><li>Exceeding 100GB? Keep in mind: </li></ul><ul><ul><li>Backup/restore/maintenance will be harder. </li></ul></ul><ul><ul><li>Use differential backup. </li></ul></ul><ul><ul><li>All sites share the same tables. Isolate large sites. </li></ul></ul><ul><ul><li>Use multiple data files </li></ul></ul><ul><ul><li>Defrag regularly. </li></ul></ul><ul><ul><li>* Your experience may vary: H/W and usage profile dependant. </li></ul></ul>
  31. 35. Large Lists – 2000 Items? <ul><li>SharePoint supports large lists, but you must carefully plan how users view the lists to prevent performance impacts. </li></ul><ul><li>For best performance , do not exceed 2,000 items per folder or view </li></ul><ul><li>Define limits on views. Use indexed columns. Take it easy on column and field counts. </li></ul>
  32. 36. SQL Memory – 4GB Enough? <ul><li>“ 4 GB is the minimum required memory, 8 GB is recommended for medium size deployments, and 16 GB and above is recommended for large deployments. ” </li></ul><ul><li>What influences the amount of RAM? </li></ul><ul><ul><li>Number and size of Content databases. </li></ul></ul><ul><ul><li>Number of concurrent requests to SQL. </li></ul></ul><ul><ul><li>Size and width of commonly used lists. </li></ul></ul><ul><li>Remember: Minimum is where we start… </li></ul>
  33. 37. SQL Data files <ul><li>Best Practices: </li></ul><ul><ul><li>Allocate TempDB on RAID 1. (or R1 variants) </li></ul></ul><ul><ul><li>Separate Data and Logs on different LUNS </li></ul></ul><ul><ul><li>Spread databases on multiple spindles </li></ul></ul><ul><ul><li>For TempDB, Create multiple data files up to the number of CPU cores. </li></ul></ul><ul><ul><li>Pre-Grow files (Autogrow as safety net) </li></ul></ul><ul><li>SharePoint 2010 supports file groups for content databases! </li></ul>
  34. 38. Identifying Disk Bottlenecks <ul><li>Perfmon </li></ul><ul><ul><li>Monitor transfer/sec for throughput trends. </li></ul></ul><ul><ul><li>Monitor Disk sec/Read / Disk sec/Write for bottlenecks. </li></ul></ul><ul><ul><li>Monitor disk Queue length for bottlenecks . </li></ul></ul><ul><li>SQL </li></ul><ul><ul><li>Select * from sys.dm_IO_virtual_file_stats(null, null) </li></ul></ul><ul><ul><li>Solution -http://www.sqlmag.com/Articles/ArticleID/96513/96513.html </li></ul></ul>
  35. 40. Lots of New SharePoint 2010 Databases
  36. 41. Large List Throttling Configurable List Throttling And Thresholds You control when and how much! List throttling controls forces end users to create more efficient views with < x number of items.
  37. 42. Web Part Performance Dashboards
  38. 43. Best Practices Analyzer Health Rules Runs on a Timer Job Create your own! Repair Auto-magically!
  39. 44. Logs & Reporting to the DB Extensibility for reporting and possibilities are limitless
  40. 45. Summary <ul><li>SQL is extremely important to SharePoint health and Performance </li></ul><ul><li>Put SQL on 64bit. (Required for SharePoint 2010) </li></ul><ul><li>SQL 2008 Enterprise – Scale, HA, compliance security features </li></ul><ul><li>Think IOPS when designing disk arrays. </li></ul><ul><li>Always separate work loads with the following priority: temp, log, search, content. </li></ul><ul><li>SQL scales up and out. Don’t push the limits upward, but keep manageability and costs in mind when scaling out. </li></ul><ul><li>Designing enterprise services with great care. Separate SSP and Search when possible. </li></ul><ul><li>SharePoint 2010 brings more databases so strategically plan for 20-50 dbs min… </li></ul>
  41. 47. © 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. Required Slide
  42. 49. Search Disk Performance Reference: http://blogs.msdn.com/enterprisesearch/archive/2008/05/19/sql-monitoring-and-i-o.aspx Drive IOPs Read (max) IOPs Write (max) Ratio Read/ Write Latency Read (sec) Latency Write (sec) Search DB Logs 14.67 1,777.29 0.01 0.3060 0.8550 Temp DB 1,110.98 1,492.01 0.74 1.6870 3.5660 Query file group 3,507.26 1,631.96 2.15 3.4360 3.2140 Crawl file group 3,043.93 371.65 8.19 15.0840 15.8720
  43. 50. Applying the Newest Learnings <ul><li>Add more processor to the backend: 4 cores to 8 cores </li></ul><ul><li>Add more RAM: 16GB to 32GB </li></ul><ul><li>Run profile sync on our terms! Run the jobs as little as possible. Once a week or once a month. </li></ul><ul><li>Separate SSP SQL instance from Search SQL instance. </li></ul>

×