Presentation from Stockholm about proper way to monitor Domino environment, out of the box rules, check values for recommended values

1. Next
Generation
Domino
Monitoring
Vladislav Tatarincev, vlaad@cyone.eu
CEO, CYONE
www.cyone.eu

2. • Domino experience since 1998.
• IBM Premier Business Partner
• Professional team:
- Certified IBM Lotus Domino Administrators (4.x,5.x,6.x,6.5,7.x,8.x,8.5.x,9.0)
- Tivoli Certified Professional (IBM Security/Tivoli Directory Integrator)
- IBM DB2, Oracle DBA
- Designed 2FA solution for IBM Verse/Domino iNotes/XPAGES www.cyone.eu/2FA
- IBM Domino Monitoring www.cyone.ru/CYMON/
- Cо 3 IBM Redbooks.
- Optimizing Domino Administration
- Domino 8.5 Deployment Guide
- Lotus Traveler Deployment Guide
- Panagenda IONET SOFTWARE
- EXTRACOMM RedHat Advanced Partner
- MobileIron Partner
- Barracuda Crossware
Company “CYONE”

3. Vladislav Tatarincev (vlaad@cyone.eu )
Blog www.vlaad.lv
• Domino/Notes Experience from 1998
• Certified IBM Domino Administrator (4.x,5.x,6.x,6.5,7.x,8.x,8.5,9.0)
• Certified IBM Domino Developer (5.x,6.x,6.5,7.x,8.x,8.5,9.0)
• Сertified IBM Instructor (6.x,6.5,7.x,8.x,8.5,9.0)
- CoAuthor 3 IBM Redbooks
- Optimizing Domino Administration
- Domino 8.5 Deployment Guide
- Lotus Traveler Deployment Guide
- Speaker on Domino/Notes usergroups:
- IBM Domino Performance Tuning
- IBM Connections Adoption - Kaizen

4. What Domino administrator should do daily?
• Maintain infrastructure, deploy new version, and new functionality
• Detect server availability and solve problem
• Help programmers and business users
• Solve user problems, grant rights , consult, process requests;
• Check AdminP (Pending Administrator Approval, Name Change Pending, errors)
• LOG.NSF (check for errors, react, solve)
• DDM.NSF – check found problems and solve them. Change server configuration
• Inetlockout.nsf (check if user is locked, analyze if mistyped password or hacker attack)
• Domlog.nsf – HTTP / Traveler /Outlook IMSMO
• Check if IBM Traveler, Outlook plugin, HTTP, SMTP, IMAP, POP3, DIIOP, LDAP
available and performs ok
• Integration check IBM Connections, Sametime, AD, TDI, DB2/Oracle
• We should participate also in projects and do “REAL WORK” 

5. Daily check of system databases 1 server – 30-60 min
• 1 Server – minimum (0,5 hour) – if everything is perfect, if not more
• 16 servers – 8 hours if everything is perfect, 32 servers - 16 hours (2
admins)
• We don’t count that administrator must:
• Learn new things, improve competence
• Have a vacation
• Can be sick
• May participate in a meetings
• Solves problems
• Share knowledge with other Admins/Colleagues
• Make changes in infrastructure, document changes, test them

6. Haddon’s Matrix
• Using Haddons ideas different countries implemented techniques to lower death toll on
roads (safety belts, penalties for speeding, slow speed areas)
• Ideas how to predict problem (limit speed,
• How to solve it faster if it has occurred (Automatic SOS, GPR coordinates)
• Lessons learned.
• Constant environment improvements, analysis of problem, solution automatization.
• Analysis of new problems.

7. I have a dream… Event Driven Administration
• Administrator goes to particular databases only when something need
to be done (inetlockout.nsf, admin4.nsf, etc)
• Increasing IT stuff to support more servers/users is one option
• Making existing IT much more efficient is better
• Since hard to find skilled people
• People come/leave
• Training needed
• People need to sleep, cant do 24x7

8. What Monitoring can do daily 24x7?
• Maintain infrastructure, deploy new version, and new functionality
• Detect server availability and solve problem
• Help programmers and business users
• Solve user problems, grant rights , consult, process requests;
• Check AdminP (Pending Administrator Approval, Name Change Pending, errors)
• LOG.NSF (check for errors, react, solve)
• DDM.NSF – check found problems and solve them. Change server configuration
• Inetlockout.nsf (check if user is locked, analyze if mistyped password or hacker attack)
• Domlog.nsf – HTTP / Traveler /Outlook IMSMO
• Check if IBM Traveler, Outlook plugin, HTTP, SMTP, IMAP, POP3, DIIOP, LDAP
available and performs ok
• Integration check IBM Connections, Sametime, AD, TDI, DB2/Oracle
• Monitor 3rd party (any database)
• Recheck important values not daily but every second/minute/hour/when needed
• Notify others on changes
• We should participate also in projects and do “REAL WORK”  / speak on user events like this

9. • Domino ˜400 metrics and stats checked
24/7
• Many Dashboards for different purposes
• Users
• Mail
• Network
• Cluster server stats
• Custom databases/3rd party
• Performance
• Traveler/IMSMO/Verse
• You can create your own
dashboards
• www.cyone.eu/CYMON/
+ CLOUD
(DAC)

10. • Ability to monitor custom / 3rd
party applications:
• Number of documents in queues
• Number of document @Formula
• Database open time
• Database search time
• CYMON can solve found issues
according to scenario, run script,
command, agent, etc.

11. Solution is built on top of opensource Zabbix
https://www.zabbix.com
Zabbix – free
OpenSource solution
Agent or Agentless monitoring
+ C API client
+ 400 Rules to understand OK/NOK

12. Automatic detection/prediction – allows to solve in advance or decrease
impact of a problem
Yellow – Warning
Red – don’t move. Problem that may be was ignored in past
We don’t check oil level, gas level or tire pression every time we
start a car, this can be “Event driven” – when needed
(Event Driven – when event occurs, real human needs to solve it)
экономит время

13. Dashboards Tesla vs Concorde
• New technologies allow bring attention to information that is needed NOW
• There are approx. 4000 statistics on Domino, they change every second
• There are databases, interfaces that don’t appear in statistics but need to be
monitored

14. Situation in IT
• Business wants projects done cheaper, faster, with less amount of
people.
• Amount of servers, volume of data, complexity of infrastructure also
grows.
• Number of systems, interfaces – also grows
• There is only one solution - Automatization

15. Event Driven Administration
• Attention of administrator is attracted in advance and only when needed. Problem
solved before it impacts business and users
• Domino and all components monitored 24х7
• Monitoring not only reports problem, but tells how to fix, thus:
• reducing time to solve problem
• Increasing knowledge of stuff about Domino
• Less skilled person can solve issue, if has instruction, more skilled people has more
time to do their work
• Monitoring can autosolve issues.
• More understanding about how Domino works

16. Example of manual work vs Automatic
Manual
• Monday 2:00 – DAOS out of sync (attachments goes to NSF without DAOS) (higher load on hdd, disk free space end faster,
bigger backups, slower server)
• Monday 9:00 Admin detects this and schedules DAOS RESYNC next night (attachments still goes to NSF, without DAOS)
• Tuesday (DAOS is ok, new attachments goes to DAOS, but Monday arrived files in NSF)
• We schedule Compact –c for weekend
• Next Monday, everything is compacted and in DAOS
• Summary : it took us 1 week to completely solve issue, risk of running out of space in case of big attachment sent to
everybody admin time around 2 hours.
Automatic:
• Monday 2:00 – DAOS out of sync
• Monday 2:01 – DAOS RESYNC launched by monitoring
• Monday 9:00 – Admin reads that there was an issue, but autosolved
• Summary: no risk, admin time almost 0, or to think “how cool to have such monitoring!”, problem resolved in 1 min

17. Domino Consultant/Tuner v.2.0 Included
• Every problem has Action what need to be done
• If somebody upgraded server, everybody will be aware

18. • A lot of rules for different Domino areas: Domino core, AMGR,
Cluster, Performance, Traveler, Users (certificates, lockouts)

19. Traveler: HTTP Threads vs device count

20. IBM Traveler и IBM IMSMO (Outlook Plugin)
1. Problem №1 - Tracking HTTP Threads vs Traveler Devices (20% rule)
2. Traveler.Status
3. Traveler.Device.Count.200 – good codes, must grow
4. Traveler.Device.Count.40х – bad codes, should not grow, should be static
5. Diagram 200 says about Traveler sync status

21. Why we need such tools?
• Monitoring allows to automate IBM Domino administration , reduces
load on Domino Admins and does it 24х7
• Checks against HCL/IBM Best Practice
• 300-400 Metrics monitored out of the box. Customer or CYONE can
add custom rules to monitor 3rd party databases, interfaces
• Monitoring can be notification only or if scenario is know, fix problem
• Monitoring keeps an eye on Domino statistics or any NSF
• Monitoring predict problems and allow us to solve them in advance,
while they dont impact business

22. What are the options to monitor Domino
• 1 – Manually when user cry  (reactive approach)
• 2 – Domino Administrator – Grandfather of monitoring
• 3 – Domino Domain Monitoring – Minimal option to monitor
Domino today
• 4 – Proactive Monitoring – Next Gen Predictive Monitoring

23. DDM –Domino Domain Monitoring (starting Domino v7)
ddm.nsf – monitors:
• Numeric values (Number) Mail.Waiting
• Tasks state (Tasks)
• Keywords in log files(Corrupted, Granted Full Access)
• Checks for other server availability
• Mail delivery tests – Mail boomerang

24. How to check if we are ok? Full Picture
• Statistics «Show stat»
• Particular Views in Domino (AdminP, inetlockout.nsf, domlog and
others)
• Statistics of OS
• Values from different system databases (Error logs, conflicts)
• Information about clients (Panagenda MarvelClient)
• Admin Toosl – Catalog, self made toosl

25. Domino v10
• Support of NewRelic (Cloud only)
• Push-out of statistics
• https://engage.ug/engage.nsf/pages/2018_Slides_e/$file/Engage2018_Domino10.pptx

26. Statistic values are 5% of full picture
• Domino has 3000 - 4000 statistics and only interpretation of values and thresholds values is
needed to make decision. Just statistics or graphs are useless!
• Border values = thresholds are very important, sometime comparison of 2-3 parameters is needed
to understand we have problem
We know that:
• Platform.DiskX.AvgQueueLen >1 – bad, if more than 2 very bad;
• Update.DeferedList >40 not good, if >200 we are in trouble
• FT.Search.Average.TimeMS <1000 ok,
more than 3000 = bad, if more than 5000 –users wait too much
for search results
• Mail.Waiting / Mail.Waiting.DNS
• Spike are hard to notice without proper monitoring

27. History of Toyota 80ies
• Company was able to predict faulty parts before goods were delivered to
client
• This increased customer satisfaction, and their cars were attractive from
safety point of view

28. In Domino problems as well as solution of these problems can be predicted
• Analysis of 404 says of missing resources, wrongly configured links/redirects,
while 404 spikes says about hack/scan attempts
• 403 or 40х HTTP codes about hack attempts
• Growing number of locked out users – symptom of bruteforce attacks
(inetlockout.nsf) (and users loosing access to system)
• Every Domino area has indicators, that it will be hard day if you don’t notice
this

29. Monitoring tracks 200-400 different statistics
• It helps to make decision, where is a bottleneck, which parameters
should be adjusted
• Nice charts helps to understand, how changes or load impacts
performance
• Integration with messengers like Telegram – 24 х 7 your control
situation

30. Example 1: monitoring of integrations or any custom NSF
• TDI solution that syncs Active Directory vs Domino
• Active Directory + IBM Domino {Create, Update, Delete, (un)lock}
1. New Requests (<20) ok,
2. Processed Requests , must grow +5 (last 12 hours)
3. Problematic Requests/Errors, must not grow
• Support of @Formula

31. Example 2: Monitoring of Admin Tools
• Panagenda MarvelClient (Users with Old client or ODS=43)
or
• Catalog.nsf (databases with Anonymous > No Access))
• Users that soon will expire and needs renew their user.id

32. Different dashboards for different users
• User – For admins and Helpdesk
• Performance
• Security
• Domino сore
• Traveler/Outlook Plugin/WebMail
• Mail
• Custom Databases
• Cluster
• Network

33. Users
• Certificate Expiration

34. Monitoring – full picture of environment

35. Recommendations how to fix problem

36. Mail

37. Management

38. Network

40. We even monitor water and order it automatically!

41. Monitoring – like Lego or Minecraft. There is no end
• Out of the Box – professional monitoring Domino/Traveler/Sametime/Verse
On Premise/
• Can monitor Domino + Business apps
• Can monitor anything (Active Directory, Switch, or water cans)

42. Features of next gen monitoring - CYMON
• Supports Domino v8.x, Domino v9, Domino v10
• Support of @Formula
• Platform statistics
• Custom database search, open time, @formulas
• Different DashBoards
• Professional recommendations how to solve problem
• Solutions can be translated to local language
• Agentless – no agent on Domino (AS400, Linux, Windows)

43. Phases of deployment
• 1 day to install – Deployment and basic configuration Domino
• Real results in 1 hour after importing and booting VM
• X weeks – Environment cleanup, solving found problems
• Y weeks – configuration of monitoring of 3rd party/ custom databases
• Z weeks – configuration of scripts to solve frequent problems

44. Deployment of Next Gen Monitoring:
• Reduce load on admin – up 60-80%
• Admin will do less routine work, will spent time on Upgrades/more creative work
• Will increase availability of system/less downtime/more profitable business
• Will allow automatically solve problems
• Will increase performance
• Will increase security
• Will increase competence of admins
• Administrator controls system, not doing blind administration or when it is already too late
• Monitoring will increase knowledge of young admins about Domino
www.cyone.eu/CYMON

45. Ultimate protection for Domino:
• Web based apps
• IBM Verse on Prem
• Compliance with #GDPR and
other security regulations
• Different ways to deliver 2nd
factor
• TOTP
• SMS
• Alternative email
• Installed in less than 10 min

46. • Domino ˜400 metrics and stats
checked 24/7
• Many Dashboards for different
purposes
• Users
• Mail
• Network
• Cluster server stats
• Custom databases/3rd party
• Performance
• Traveler/IMSMO/Verse
• You can create your own
dashboards

47. • Ability to monitor custom / 3rd
party applications:
• Number of documents in queues
• Number of document @Formula
• Database open time
• Database search time
• CYMON can solve found issues
according to scenario, run
script, command, agent, etc.

48. #WeLoveDomino – Free version of both Product
#2FAforDomino Free edition –
You can protect 5 users:
• VIP accounts or
• Admin or system accounts
• More info www.cyone.eu/2FA
• Soon on every Domino server
Starting 01 July 2019
#CYMON – CYONE Monitoring
• Users and Mail modules
• More info www.cyone.eu/CYMON
• Available 01 July 2019

49. Thank You
www.cyone.eu