1. Notes from the Field
Presented to the Swiss PASS Chapter - Zurich.
Hotel Continental - Lausanne, Switzerland
15:30 – 19:00, June 9th 2011
Charley Hanania, QS2 AG
B.Sc (Computing), MCP, MCDBA, MCITP, MCTS, MCT, Microsoft MVP: SQL Server
Senior Database Specialist

2. My Background
• Now:
• Microsoft MVP: SQL Server
• Database Consultant at QS2 AG
• Formerly:
• Production Product Owner of MS SQL Server Platform at UBS Investment Bank
• Technical Team Lead
• Consultant etc…
• ITIL v3 Certified
• SQL Server Certified since 1998
• On SQL Server since 1995
• Version 4 on OS/2
• IT Professional since 1992
• PASS
• Chapter Leader – Switzerland
• Regional Mentor – Europe
• 24 Hours of PASS Team Member
• Event Speaker
3
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

3. Contact Info
• Email: Charley.Hanania@sqlpass.org
• Website: http://www.sqlpass.ch <- Slides here
• Twitter: http://www.twitter.com/CharleyHanania
• Blog: http://blogs.mssqltips.com/blogs/charleyhanania
• Linked-in: http://www.linkedin.com/in/charleyhanania
4
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

4. Agenda
• Best Practices Defined
• “Some” Operational Best Practices
• Best Practices in this Presentation
1. Obfuscation
2. Rubber Stamping your installation
3. Using Projects and Solutions
4. Governance
5. SLA’s, SLE’s, SLO’s
6. Monitoring Appropriately
7. Diagnostics Reports
• Recap
• Questions
5
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

5. Definition :: Best Practices
• Focus SQL Server / Operations
• Guidelines, methods, procedures & policies that have been used
with numerous customers & installations and have assisted with:
• Root Cause Analysis
• Clarity of Design
• Avoidance of Issues
• Performance
• Security
• Disaster
• Speed of implementation
• System Longevity
6
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

6. “Some” Operational Best Practices
7
Rename SA
Rename Administrator
Disable SA
SQL IO
64KB Cluster Size
Multiple Files/Filegroups
Multiple TempDB Files
Partitioning
Filtered Indexes
Development Standards
Projects and Solutions
Scripting with SQLCMD
Scripting and Automating with Powershell
Multiple development Environments
Singleton Development Environments
Principle of Least Privilege
Governance through PBM
Security Engineering
Performance Engineering
Resilience Engineering
Backup/Recovery
Compression
Solid State Disks
Recovery Planning
Availability Planning
Disaster Planning
Monitoring
Service Level Agreements
Service Level Expectations
Service Level Objectives
Templated Installations
Capacity Planning
Check Sheets
Run Books
Encryption
Authentication Certificates
Code Reuse
Plan Reuse
indexing for Read
Indexing for write
Asynchronous logging & messaging
Virtualisation
Maintenance Checking
Consolidation
Baselines
Instances
Port Changes
SQLServerOperationalBestPractices-NotesfromtheField
Page File
Instant Initialisation
Memory

7. Best Practices in this Presentation…
8
Rename SA
Rename Administrator
Disable SA
SQL IO
64KB Cluster Size
Multiple Files/Filegroups
Multiple TempDB Files
Partitioning
Filtered Indexes
Development Standards
Projects and Solutions
Scripting with SQLCMD
Scripting and Automating with Powershell
Multiple development Environments
Singleton Development Environments
Principle of Least Privilege
Governance through PBM
Security Engineering
Performance Engineering
Resilience Engineering
Backup/Recovery
Compression
Solid State Disks
Recovery Planning
Availability Planning
Disaster Planning
Monitoring
Service Level Agreements
Service Level Expectations
Service Level Objectives
Templated Installations
Capacity Planning
Check Sheets
Run Books
Encryption
Authentication Certificates
Code Reuse
Plan Reuse
indexing for Read
Indexing for write
Asynchronous logging & messaging
Virtualisation
Maintenance Checking
Consolidation
Baselines
Instances
Port Changes 07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField
Page File
Instant Initialisation
Memory

8. Obfuscation 9
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

9. Obfuscation
• Why?
• Security…
• If you can’t find the lock you can’t try your key
• If you can’t find the door knob, you can’t open the door
• If you can’t find the door you can’t get in nor out.
10
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

10. Obfuscation
• Customer Scenarios
• Requirement:
• Build me a Laptop Demo copy of our offering to Military Security Standards
• If lost / stolen its useless
• Valid Intellectual property
• Full suite of software, middleware, back-end
• Fully self-contained
11
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

11. Obfuscation
• Customer Scenarios
• IP Conscious Customer:
• Make sure our data and databases are completely secure.
• When you’re done, we’ll contract a Russian hacker to break in.
12
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

12. Obfuscation
• Solutions:
• Hide…
• Change Default SQL Instance
• Change Default SQL Ports
• Change SA Password
• Rename SA Account
• Rename Administrator Account
• Change Key Windows Ports
• eg. RDP
• Multi-level Firewalls
• Encryption
• Manual / Physical Policies of protection 13
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

13. 14
SQLServerOperationalBestPractices-NotesfromtheField
07 June, 2011
Some Obfuscation Techniques

14. Rubber Stamping your Installation 15
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

15. Rubber Stamping your installation
• Why?
• Closer to the single version of the truth.
• Know the parameters your system operates under.
• Rebuild, reset, replace.
• Speed of deployment.
16
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

16. Rubber Stamping your installation
• Customer Scenarios
• Development / Test / Production Environments were out of Sync
• Collations
• Service Accounts
• File locations
• Other Environmental Issues
• Changes developed and tested failed or performed badly in Production
17
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

17. Rubber Stamping your installation
• Solutions:
• Unattended Installation Files
• Step-By-Step documentation
• Slip-streamed installations
• Scripted Installations
18
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

18. 19
SQLServerOperationalBestPractices-NotesfromtheField
07 June, 2011
Creating an Unattended Installation File

19. Using Projects and Solutions 20
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

20. Using Projects and Solutions
• Why?
• Gives order to implementations
• Grouping of items for a solution
• Search and reuse
• Assists in Team-Oriented approaches
21
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

21. Using Projects and Solutions
• Customer Scenarios
• Teams of 3+ developers
• Overwriting each other’s code
• Loss of scripts and object definitions
• Propagation of the wrong objects to test/production
• Unclear path to implementation
22
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

22. Using Projects and Solutions
• Solutions:
• Use Ticketing system for functionality requests
• Agree on scope of work
• Develop with clear, sequenced file naming – even scratch files
• When ready for promotion into communal development environment,
group files into a “Release xxx” project.
• Deploy, test etc…
23
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

23. 24
SQLServerOperationalBestPractices-NotesfromtheField
07 June, 2011
Using Projects and Solutions

24. SLA’s, SLE’s, SLO’s 25
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

25. SLA’s, SLE’s, SLO’s
• What?
• SLA: “Service Level Agreement”
• Agreement with Business / Management on what the
applications/services/systems will provide to the “end customer”
• SLE: “Service Level Expectation”
• Agreement of mutual understanding with Business / Management on what the
applications/services/systems will provide to the “end customer”
• SLO: “Service Level Objective”
• Agreement internally on what the applications/services/systems will provide to
the “end customer”
• Focus:
• Performance
• Availability
• Recoverability
• [Not Functionality]
26
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

26. SLA’s, SLE’s, SLO’s
• Why?
• Without agreed, contracted and planned service levels, management
and customers (users) have nothing to track the:
• Availability of Service;
• Performance of Systems; and
• Quality of Implementation.
• Management can assign funds, plan and allocate relevant resources to
meet the Agreements
• Business understands the true process, business and technical
dependencies of customer facing systems.
• Change is entered into warily and with full support of all associated
parties.
27
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

27. SLA’s, SLE’s, SLO’s
• Customer Scenarios
• 1 Resource :: Developer/DBA/DB{system} Admin for a customer critical
system across APAC and Europe.
• Management were happy:
• Simple Change Management
• Quick Feature implementation
• Low Costs
• High Revenues
• Resource was happy:
• Job Stability
• Developer Artistic Licence
• Integral Part of the Team
28
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

28. SLA’s, SLE’s, SLO’s
• Customer Scenarios
• 1 Resource :: Developer/DBA/DB{system} Admin for a customer critical
system across APAC and Europe.
• Problem:
• Management and Developer had an argument on strategy
• Productivity ebbed
• Quality degraded
• Motivation decreased
• Resource was “sick” more often
• All Application and system IP was with the 1 Resource!
29
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

29. SLA’s, SLE’s, SLO’s
• Customer Scenarios
• Very technical team
• Eat and Breath Technology
• Communication skills – average
• Management adapted to Technical Team style
• Unmanaged changes
• Periods of instability
• Overall - best efforts support
• Operations Expanded
• Customer base increased
• Internal and external customers became highly dependant on the
technology base. 30
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

30. SLA’s, SLE’s, SLO’s
• Customer Scenarios
• Very technical team
• Eat and Breath Technology
• Communication skills – average
• Periods of system unresponsiveness
• Outages affecting the whole organisation
• etc
• Periods when users “felt” like there were issues with the application.
31
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

31. SLA’s, SLE’s, SLO’s
• Solutions:
1. Define which applications, systems and services are in production
2. Define and categorise them by important to business
• Including cost & revenue
3. Define metrics focussed on business processes such as:
• Customer Creation
• Business Transaction completion
• Business Transaction retrieval
4. Define Capacity / Availability needs such as:
• X New Customers per day
• Y Business Transactions per hour
• Available during core business hours (8x5) etc
32
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

32. SLA’s, SLE’s, SLO’s
• Solutions:
5. Review Platform abilities based on Definitions
• Measure Current as a baseline
6. Review Human Resource needs based on defined System Availability
and Capacity needs.
7. Define a plan for when/if system can approach those levels
• Redevelopment / refactoring
• Personnel Acquisition / Acquiescence
8. Measure and Monitor Progress
• Stops the fire-fighting
• Allows partners and customers to see progress
• Gives agility in approach, focussed on importance and need.
33
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

33. Monitoring Appropriately 34
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

34. Monitoring Appropriately
• Why?
• If you don’t know, then… You don’t know…
• Availability
• Capacity
• Performance
• Recoverability
• Reduces Fire-fighting
• Leaves your fingers on the Pulse.
35
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

35. Monitoring Appropriately
• Customer Scenarios
• Too Little:
• “Why is my system slow…?”
• “Was the system up last night? I tried to…”
• How long would it take us to rebuild
• Too much:
• “We missed the performance issue because we had thousands of Printer
Job Completion entries to close…”
• “We can’t add any more SQL Counters as there’s no more capacity in the
DW”
• “Why’s the Network so slow? Oh, its all the traffic being sent to the
Monitoring and alerts DW…”
36
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

36. Monitoring Appropriately
• Solutions:
• Centralised “System Monitor”-Type Consoles
• Environment at a glance
• Standard interfaces
• Alerts and trending
• Agent or Poll methodologies
• Technologies:
• Systems Centre Operations Manager
• Nagios etc
• System Monitor
• Management Data Warehouse
• DMVStats
• SQL Server 2008 Management Studio
• Activity Monitor/Performance Dashboard Reports
• SQL Server Health and History Tool
• RML Utilities for SQL Server
etc
37
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

37. Governance 38
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

38. Governance
• Why?
• Governance is simply the execution of standards policies
• For control
• For consistency
• For Reporting
• For ease of Support and Maintenance
39
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

39. Governance
• Customer Scenarios
• Too much:
• Large Organisation
• Such strict policies that everything was either unsupported or an exception.
• Reduced Productivity
• No policy on items that weren’t covered by the policies.
• Too little:
• Large Organisation
• Do what’s needed to fulfil the requirements
• Support nightmare
• Great for consultants! 
• No Standards / documentation either
• SLA’s ??? 40
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

40. Governance
• Solutions:
• Various frameworks eg. ITIL
• For SQL Server Auditing / Control:
• DDL Triggers
• Policy Based Management
• SQL Server Auditing Framework
41
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

41. 42
SQLServerOperationalBestPractices-NotesfromtheField
07 June, 2011
Using Policy Based Management

42. 43
SQLServerOperationalBestPractices-NotesfromtheField
07 June, 2011
Introduction to SQL Server Audits

43. Diagnostics Reports 57
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

44. Diagnostics Reports
• Why?
• As a tool to help when investigating issues
• Is a cursory finger on the pulse
• Should be 2-pronged
1. Business focussed
2. System focussed
58
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

45. Diagnostics Reports
• Customer Scenarios
• System with thousands of devices reporting on short intervals and
bursts (status etc) asynchronously
• Business processes triggered on certain criteria
• Customer facing interfaces experience varying loads
• Database is core to the offering
• When looking at issues the general same approach is taken
1. Are the feeds running?
2. Are they coming through with the expected throughput/latency?
3. Are major process components locking / blocking?
4. Are the External Interfaces open and serving?
59
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

46. Diagnostics Reports
• Solutions:
1. Use scripting languages to provide a “printer test page”-type report
2. Use Reporting Services to provide a consolidated view of several
systems
3. Use SQL Server Management Studio’s Custom Reports to get instance
focussed info or to report out from a warehouse.
60
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

47. Recap
• Best Practices Defined
• “Some” Operational Best Practices
• Best Practices in this Presentation
1. Obfuscation
2. Rubber Stamping your installation
3. Using Projects and Solutions
4. Governance
5. SLA’s, SLE’s, SLO’s
6. Monitoring Appropriately
7. Diagnostics Reports
• Recap
• Questions
61
07 June, 2011
SQLServerOperationalBestPractices-NotesfromtheField

48. Questions?
SQLServerOperationalBestPractices-NotesfromtheField
62
07 June, 2011

49. Now: Refreshments & Networking