Preventing Server Sickness Becoming A Pandemic - Benelux March 2013

1. Prevent Server Sickness Becoming a
Pandemic!
Gabriella Davis
The Turtle Partnership
gabriella@turtlepartnership.com
twitter: gabturtle

2. Fixing Your Server
What causes server sickness
Tools to spot sickness
Getting Your Server Back to Full Health
2

3. Server Sickness
3

4. Server Sickness
The problem with Domino
How does a server get sick?
–Vulnerabilities
–Aging Configurations
–Bad Habits
–Developers Gone Wild
4

5. The Problem With Domino
“My Server Is Running Fine”
Server Stability
–Often despite our best efforts
Tasks that just run
–even without being properly configured
5

6. Vulnerabilities
Start with the OS
–patch levels
–unnecessary processes with exposed ports
–disk and data security
Then the hardware
–It’s all about disk performance
–Using a SAN? Is the SAN configured for Domino?
–Transaction logs configured?
6

7. Vulnerabilities
Security
–ACLs
• -Default- and Anonymous
• LocalDomainServers
HTTP vs HTTPs
LDAP
DIIOP
Sametime
7

8. Aging Configurations
What can give you problems over time
–Database sizes
–More users
–More tasks and features
8

9. Bad Habits
What are your users doing?
–what features are they using
–how are they using them
• are they creating repeating 10yr appointments for instance
• are they copying themselves on emails
Password quality for HTTP passwords
9

10. Giving Developers Power
Allowing development to dictate replication and
agent scheduling
The curse of not production tested XPages code
Demands for “LDAP” or “DIIOP” for an
application to work
10

11. Tools to Spot Sickness
11

12. Tools to Spot Sickness
Understanding Priorities
DDM Probes and Event Analysis
Statistics
Catalog.nsf
QoS - new with Domino 9
Enhanced Fault Reporting - new with Domino 9
12

13. Understanding Priorities
Server role
–What do you want from your server
–What are statistics telling you
Warning Levels
–Is it safe to ignore ‘Warning (Low)’ and focus on
‘Fatal’ or ‘Failure’
13

14. Bringing Problems to You
Event Handlers, Event Generators, Statistics, Fault
Reports and DDM Probes - where to start
Setting Statistic Thresholds
Choosing and configuring probes
Reviewing Faults
Setting up QoS behaviour
14

15. Bringing Problems To You
Why we set up collection hierarchies for DDM
–and how
Daily and Weekly DDM reviews
–What to look out for
15

16. Probes for Mail Servers
Security - Weekly
Directory Performance
Critical mail routes
Mail ‘Slack’
16

17. Probes for Application Servers
Agent run times
–agent cpu usage
Security and Web Configuration
17

18. Probes for Struggling Servers
OS level
–disk performance (beware of reported SAN
problems)
–memory
–network
18

19. What to look for
Fatal problems
Persistent Warnings
Peak activity behaviour
–uptick in problems at 9am, 1pm etc
Repetitive low level ‘annoyances’
19

20. Catalog.nsf
Not every database is immediately visible but they
are all there (just hidden with selection formulae)
It’s a good place to start looking for multiple
replica
It’s a good place to find ACL issues
Replicates around your domain and updates
overnight
20

21. QoS - Quality of Service
Monitor server health and performance
Monitors application behavior, stability and hangs
Restarts Domino if it thinks there are memory issues
or an application is hung
Shuts down Domino if a clean shutdown doesn’t
happen and the server hangs
Controlled via notes.ini settings and dcontroller.ini
Requires Domino to be running under the Java
Controller
• nserver -jc 21

22. QoS Configuration
Starting Domino under Java Controller should
create a dcontroller.ini file
QOS_Enable=1
In Notes.Ini
• QOS_ProbeInterval (defaults to 1 min)
• QOS_ProbeTimeout (defaults to 5 mins)
• QOS_ShutDown_Timeout
• QOS_Apps_Timeout
• QOS_Shutdown_Timeout
22

23. QOS - Potential Problems
QOS doesn’t support passwords on server ids , the
restart will pause at the password entry screen
QOS timeouts being too low
Don’t enable QOS on servers without transaction
logging
23

24. Enhanced Fault Reporting
Fault Reporting Database -lndfr.nsf
Expanded to include a by Disposition view
–all faults when analyzed have a disposition value
that categorises as
• Problem
• Possible Problem (possibly actionable )
• Possible Problem (likely NOT actionable )
• Informational
• Unknown (investigate)
24

25. Possible Problem - Actionable
Out Of Memory: Represents a crash in which the Java virtual
machine (JVM) ran out of a memory resource such as heap
space.
Launched Notes multiple times: Indicates that the user
quickly launched multiple instances of the Notes client
Possible hang: Indicates that the Notes client was manually
terminated while it appeared to be doing useful work.
User Kill: Indicates that the user manually terminated the client
while it appeared to be waiting for input or network timeout
25

26. Back to Full Health
Getting Control
–Mail , Databases and ECLs
–SMTP
–Agent Scheduling
–Directories
–Adminp
–LDAP
–Tasks and Internet Site Documents
Domino Configuration Tuner
26

27. Back to Full Health
Getting Control
–Mail , Databases and ECLs
–SMTP
–Agent Scheduling
–Directories
–Adminp
–LDAP
–Tasks and Internet Site Documents
Domino Configuration Tuner
27

28. Getting Control
Mail and Databases
Setting ACLs at directory level (Editor)
Lock down ECLs via Policies
Introducing quotas alongside server based archiving
Consider archiving files to a dedicated server
Upgrade to 8 and enable OOO router instead of
agents
Disable forwarding rules set up by users
Use message tracking and mail rules very sparingly
Disable on the fly searching of non indexed
databases 28

29. Database Management Tools
DBMT Server Command
• runs copy-style compact operations
• purges deletion stubs
• expires soft deleted entries
• updates views
• reorganizes folders
• merges full-text indexes
• updates unread lists
• ensures that critical views are created for failover
–Replaces Updall
• Load updall - nodbmt tells updall to run but not perform the
functions that DMBT already does
29

30. DBMT Parameters
-compactThreads
-updallThreads
-ftiThreads
-timeLimit refers to compact timeout for DBMT
-range starttime stoptime
–compactNdays (run Compact every x days)
–ftiNdays (run FT Index every x days)
–force d (day Sunday =1) fixup if compact fails for
consecutive day
30

31. Getting Control
SMTP
Restrict relaying to specific ip addresses not
network ranges
Beware of allowing authenticated relaying and
opening up to dictionary attacks
Restrict rights to send to internal groups from
internet addresses
Don’t accept mail for local part matches
Configure your server for HTML mail not plain
text
31

32. Getting Control
SMTP (more)
Don’t allow all connecting hosts to deliver mail
inbound, if you use a service restrict to those hosts
Use services / tools to spot attacks such as
–persistent attempts to mass deliver within a time
period
–continual failures by a host to deliver to a correct
address
Move responsibility for that first line of defense
away from native Domino
32

33. Getting Control
Agent Scheduling
When are agents set to run
–amgr_newmaileventdelay
–amgr_newmailagentmininterval
If you’re using OOO agents how often are they
scheduled
Do users have private agents running
–Sh Agents [DBName]
• All shared and private agents in a database
Who has rights to run agents
33

34. Getting Control
Directories
Avoid adding additional views to the Domino
Directory
The risk of allowing local replicas with Author
rights
Directory Assistance
–Sh xdir
34

35. Getting Control
Adminp
Purge old documents
Requests awaiting approval
Tell adminp process NEW not ALL
35

36. Getting Control
LDAP
Allowing anonymous access to query LDAP
Authenticating LDAP queries
Extended Directory Catalog used by LDAP
Relying on DNS
Not configuring the LDAP task correctly to allow
large searches with no timeouts
Maintaining schema.nsf
36

37. Getting Control
Tasks and Program Documents
Disable tasks you don’t need
Schedule overnight tasks so they don’t overlap
–and don’t conflict with backups
Use program documents so you can review and
manage easily
–sh config servertasksat*
Keeping templates on every server
Using compact -B
37

38. Getting Control
Internet Site Documents
Web Configuration means TCPIP tasks are
configured in the server document and are server
wide
–often enabled by default
Internet site documents require you to opt in for
TCPIP services
–configured by hostname
38

39. Domino Configuration Tuner
Domino Configuration Tuner is an analysis tool
based on a set of pre-configured best practice/worst
practice rules
The Rules are shipped by IBM with the Lotus
installs and are updated via a public update site
Makes recommendations on configuration changes
to enhance performance and security and reduce
TCO
39

40. How does it work?
Run and installed via the Domino Configuration
Tuner database
Updated by online template updates and rule
updates
DCT rules and results are held in a local database
and will require a restart of the client for changes
to take effect
Scans
–Server documents
–notes.ini settings
–advanced database properties
Intended to scan servers in a single domain
40

41. How does it work?
Creates reports on each scanned server based on
the rules you select
Each report contains
–Issues
–recommendations for adjustments
–links to supporting documentation
41

42. Pre-requisites
v8 Notes client (standard or basic) or administrator
dct.nsf database and dct.ntf template
servers 7.x or higher
42

43. Setup
DCT.NSF
StdDominoConfigTuner Template (dct.ntf)
ID must have reader access to names.nsf
ID must have ‘View Administrator’ rights
Requires no server or domain changes
43

44. View Administrator Rights
Server Document
Security Tab
View Administrator is a subset
of ‘Administrator’ rights
Think of it as ‘Show’ not ‘Tell’
rights
–Sh users - YES
–tell http refresh - NO
44

45. DCT Preferences
List of all rules
Review rule , description and supporting
documentation
All rules are enabled by default for all scans
Enable and Disable rules
45

46. DCT Updates
Connects to the IBM site to download
–must have outbound connectivity
46

47. DCT Updates
Click ‘check for updates’
Connects to an external IBM site to identifies
any template or rule updates
47

48. DCT Updates
Accept license and updates download
It’s not possible to selectively download
48

49. DCT Updates - Finished
“Successful” screen will notify you to restart
your client
You may need to do 2 client restarts before
DCT can be used
49

50. Running the tuner
First select the servers in your current domain you want
to run against
The list of servers is retrieved from the domain of the
home server identified in your location document
Change locations to scan a different domain
50

51. Running the tuner
You can manually type in the full hierarchical
names of any other servers you want to scan as
part of this analysis
Separate multiple server names with commas,
semi colons or new lines
You can only scan servers you can reach so you
need a connection document to any you list
–or the server needs to be available via your
passthru server in your location
51

52. Understanding the Results
Summary results
Issues by criticality
52

53. Understanding the Results
Summary results
Servers that failed to scan
–reason why scan failed
53

54. Understanding the Results
Summary results
Detailed list of rules evaluated
54

55. Understanding the Results
View the current report
Select ‘change’ to view a different report
55

56. Understanding the Results
Filter results to make analysis easier
–by server
–by specific rules
–by severity
56

57. Understanding the results
Categorised results of recommendations
Sorted by criticality and then by server name
57

58. Understanding the results
Each recommendation comes with an
explanation so you can evaluate on a result by
result basis if you want to make the change
58

59. Understanding the results
Each recommendation is provided with a link to
a best / worst practices supporting
documentation
59

60. Working with Rules
Disabling and enabling rules can be done
through the ‘Preferences’
60

61. Working with Rules
Selecting a rule shows the description and links
to the best / worst practice documentation
61

62. Making Changes
Advanced Database Properties
–assigned en masse via Domino Admin
notes.ini settings
–assigned via the command set config xxx = x
–shown via the command sh config xxx = x
Many recommendations refer to ‘some databases’
but don’t specify which ones - check which ones
will be affected
62

63. Resources
Domino Configuration Tuner blog
–http://www.bleedyellow.com/blogs/DCT/
–details and explanations of new rules published
each month
63

64. Summary
• No matter how well your servers are configured they will continue to
degrade in performance over time unless you pro-actively monitor and fix
• Many of the server performance issues will be seen first by your users
before they filter down to you
• Make reviewing your server configuration using DDM probes followed by
a DCT analysis part of every server upgrade
• Enable probes that are specific to the server role. Mail and Directory
probes on Mail servers and Agent probes on Application servers
• Use Security and Database probes configured in DDM to stay on top of any
low level warnings that could cause larger problems in the future
• Don’t over configure your servers to monitor everything or you’ll be
looking for a needle in a haystack. Ask your servers to tell you only what
you need to be aware of so immediately
• Use the built in tools, DCT, Statistics, DDM, Catalog, Activity Trends to
monitor your servers and gain a good understanding of what is their
‘normal’ behaviour so you can more easily spot when something goes
wrong.

65. Questions
How to contact me:
Gabriella Davis
gabriella@turtlepartnership.com
Twitter: gabturtle