RNUG - Dirty Secrets of the Notes Client

Make Your Data Work For You
Dirty Secrets
of the Notes Client
Christoph Adler
9th October 2019

Christoph Adler - Senior Consultant at panagenda
18+ years of Lotus / IBM / HCL solutions experience
Focused on
• Notes Client Management
• Analysis and Optimization of Notes/Domino infrastructures
• Application Modernization
Good to know
• German
• Bass player
• Husband, Geek, likes Music, Football (soccer) and Beer

https://www.slideshare.net/christophadler50

1. IBM/HCL Notes
2. MarvelClient
3. Upgrades - Notes 10.0.1 FP3 & Notes V11 Beta-1
4. Citrix / VDI
5. Troubleshooting
6. Performance
7. Q & A
Agenda

IBM/HCL Notes

IBM Notes → HCL Notes
• V9.0.1 (FP10 + IF7 + JVM + ST9.0.1 FP1 HF1) & V10.0.1 (FP3)
– HCL Notes V11 Beta-1
• Standard, Basic, Admin- & Domino Designer-client
• ICAA 2.0.1
• Single & Multi-User

IBM Notes – Directories (9.0.1 & 10.0.1) → old
• Program directory
C:Program Files(x86)IBMNotes
• Shared data directory
C:ProgramDataIBMNotesDatashared
• Data directory
C:Users<USERNAME>AppDataLocalIBMNotesData

HCL Notes – Directories (V11 Beta-1) → new
• Program directory
C:Program Files(x86)HCLNotes
• Shared data directory
C:ProgramDataHCLNotesDatashared
• Data directory
C:Users<USERNAME>AppDataLocalHCLNotesData

HCL Notes – V11 Beta-1
• 23.100 files in Notes Programframework in 2.000 subdirectories
– 9.000 (largely undocumented) .properties files
– 1.000 .jar files
– 600 .xml files
– 1.200 .html files
• 445 files in Notes Dataworkspace in 630 subdirectories
– 40 (largely undocumented) XML files
– 40 (largely undocumented) .properties files
• Which is
– >23.500 files in >2.600 subdirectories
– >9.000 largely undocumented files
– ... and >30 NSFs/NTFs/NDKs
– ... and the good old notes.ini file

HCL Notes – V11 Beta-1
Workspace
(nsf) Physical nsfs 1, 2, 3, … n

MarvelClient

Two Server-side Databases
• MarvelClient Config
• MarvelClient Analyze
Single binary on the Notes Client
• Deployed to the Notes data directory
• Activated via EXTMGR_ADDINS in notes.ini
MarvelClient Architecture
Config
DB
Analyze
DB
Avg.
20-70 KB
Upload
Avg.
1-4 KB
Download

• 1,000 helpdesk calls remain 1,000 management incidents
• Repeated efforts lead to limited, over time decreasing benefits
➢ Only reduces time spent on each helpdesk call
Audit Configure Benefit
Time
Classic Helpdesk client management

… solve helpdesk incidents automatically
• Actions are permanently enforced
• Benefits increase over time and can reduce helpdesk calls by up to 80% (easily)
➢ Reduces number of helpdesk calls AND
time spent on each call
(for users and helpdesk agent)
Audit Configure Benefit
Time
MarvelClient - Centrally managed actions…

Notes → Self-Healing
Live

MarvelClient Essentials – included in IBM and HCL Notes/Domino
• Makes it easier, faster and cheaper to upgrade to HCL and IBM Notes 10.x / 11.x
• Provides valuable insights into Notes client landscapes for Notes 9.x and beyond
• Helps reduce helpdesk tickets, support, client maintenance and upgrade efforts
• Runs on all Notes client platforms supported by IBM (Intel Mac 64 bit, Windows,
Citrix and VDI)
• Runs on all IBM Domino platforms supported by IBM
• Best of all: it’s FREE and included in IBM Notes and Domino 10.0.1 and higher
• If you are not on IBM Notes and/or Domino 10.0.1 (or higher) now, you can get
MarvelClient Essentials here:
https://www.panagenda.com/essentials/

• MarvelClient Essentials HCL and IBM Notes management solution
https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0036369
• Getting Started with MarvelClient Essentials
https://hclpnpsupport.service-
now.com/sys_attachment.do?sysparm_referring_url=tear_off&view=true&sys_id=ed25
7dd41b65330077761fc58d4bcb25

MarvelClient Upgrade Free
www.panagenda.com/free-upgrade

MarvelClient Upgrade Free
• Please register here to get the free license:
https://www.panagenda.com/products/marvelclient/free-upgrade/
• Documentation:
https://img.panagenda.com/download/MarvelClient/QuickStartGuide_MarvelClient-Upgrade-free.pdf
• Taking Your First Steps with MarvelClient Upgrade
https://www.panagenda.com/2019/03/taking-your-first-steps-with-marvelclient-upgrade/
• Getting the Most From Your MarvelClient Upgrade
https://www.panagenda.com/2019/03/getting-the-most-from-your-marvelclient-upgrade/

Under the Hood of Notes Client Upgrades
https://www.panagenda.com/download/MarvelClient/Under-the-Hood-of-Notes-Client-Upgrades.pdf

Upgrades
Notes 10.0.1 FP3

What we see out there in Notes deployments
• Lack of visibility and control
• Many different versions and many different installation types
• Uncontrolled configurations (Data Dirs, Locations, Connections, Accounts,
Settings, Eclipse preferences, …)
• Not secure (JVM vulnerabilities, Code Signers, ECL Management)
• Mixed functionality – things don’t work!
• The same end-user problems occur again and again (and again)

What we see out there in Notes deployments (2)
• Reluctance to upgrade
• Software deployment teams have little/no knowledge or “care factor” about
Notes
• Administrators lack time and skill to utilize all available tools to effectively
manage the Notes client
• Perception → it's too difficult and/or expensive / too risky for the business
due to disruptions

Know your current environment
• It’s fundamentally important that your know the details about your current
Notes Client Infrastructure to be able to make smart decisions for future
deployments
– Which exact versions are deployed?
– Which Installation types and modes are used?
– Locations of Program-, Data- and SharedData-Dir?
– Usage of Laptops, Desktops, Citrix and VDI in numbers/percentage?
– Which OS incl. version is used?
– Usage of Notes addons → Connections plugins, SwiftFile, CTI, … ?
• If you can’t answer the questions above 100% correctly, the following four
slides will help you a lot!

MarvelClient Essentials – AnalyzeDB examples

MarvelClient Essentials – AnalyzeDB examples (2)

Define target (version and more …)
• Version is very easy at the moment → 10.0.1 FP3
– Notes 11 will be shipped in Q4 2019
• Which Client Package?
– Standard, Basic and/or ICAA → CAA (Client Application Access)
• Which Installation type?
– Single or MultiUser

Define target (version and more …) (2)
• Adding languages using MUI-Pack installation(s)
– Multilingual User Interface
– Group1, 2a, 2b and 3 → Group1 is available (unfortunately no Russian language)

Notes/Domino v11 Language Kits
• Re-engineering language kit generation process
• Selected language kits will be available for betas
• Simultaneous ship language kits with Domino v11
• Ship language kit fix packs as necessary

• Where to deploy?
– Laptops, Desktops, Citrix and/or VDI
– Windows7, Windows10, macOS 10.13 – 10.15
https://www-01.ibm.com/support/docview.wss?uid=ibm10737113
(couldn’t find this at https://support.hcltechsw.com)

• Adding other functions/packages?
– Sametime embedded → not needed, 10.0.1 is up-to-date
– JVMPatch → not needed, 10.0.1 is up-to-date
– SwiftFile → use latest version 4.2.1
− IBM Connections Plugins for Notes → use latest version 5.5.0.20180813-1729
− Programs/Applications who interacts/communicates with Notes
CTI, Instant Messengers, …

• Let’s assume (for this session) you want to Upgrade from Notes 9.0.1 FPx IFx to the following
components:
– IBM Notes 10.0.1 Standard Client → MultiUser Installation on Windows10 64Bit
– Add French and Japanese (Multilingual User Interface = MUI)
– Fix Pack 3
– SwiftFile 4.2.1
– Connections Plugins for Notes 5.5.0.20180813-1729
– Use of default paths for
Program directory
C:Program Files(x86)IBMNotes
Shared data directory
C:ProgramDataIBMNotesDatashared
Data directory
C:Users<USERNAME>AppDataLocalIBMNotesData

Choose Upgrade/Deployment Tool – Third-Party-Solutions
• Third-Party-Solution for Notes → Today: Marvel Client Upgrade Free
• Third-Party-Solutions for Windows, like
– SCCM
– Empirum
– LanDesk
– DSM
– others

Choose Upgrade/Deployment Tool – Notes Smart Upgrade
For many reasons
I “personally” do not recommend to use “Notes Smart Upgrade”

Create Upgrade Package
• No need to make changes to MSI package(s)
– Scripts can help here
– Differentiation between package and configuration (notes.ini, ConfigFile, etc.)
– That will help you a lot when it comes to package changes, new versions and testing
• Always include an Uninstall part in your package
– In-place upgrades are not recommended
– You can use the package also in cases of a new or re-install

Create Upgrade Package (2)
Installation order
a) Uninstall current version
• Using NICE (Notes Install Cleanup Executable)
https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0040768
b) Run local cleanup
• Remove old Program- and SharedData-Dir
c) Install IBM Notes 10.0.1
d) Add French and Japanese languages (Multilingual User Interface = MUI)
e) Install Fix Pack 3
f) Install SwiftFile 4.2.1
g) Install Connections Plugins for Notes 5.5.0.20180813-1729
h) Copy customized files like stub notes.ini, ConfigFile, etc.

• Command lines for silent Uninstall and Cleanup
• Command lines for silent Install/Upgrade
• Command lines to copy customized stub notes.ini & ConfigFile

• Check the log files (created during installation) for errors for all installed
components
– Successful
– Not successful (with errors) – new installation needed after Windows restart
(complete package)

• Additional Information
– For an installation on Citrix, please check out this slideshare:
• https://www.slideshare.net/christophadler50/panagenda-webinar-virtual-faster-better-how-to-
virtualize-ibm-notesv10
• Or come to my session at RNUG tomorrow ☺
– If you need/want to switch from Single to MultiUser
• Check out the MUMA Tool (Multi-User Migration Assistant) – must be done before Upgrade to
Notes 10.0.1
– https://www-01.ibm.com/support/docview.wss?uid=swg21459627
– A specific Third-Party solution can help here during Upgrade process – also if you want
to migrate the DataDir from NetworkDrive to default local → Marvel Client Upgrade ☺

• Additional Information
– Customized stub notes.ini (example)
– config.txt (example)

Define post Upgrade actions
• notes.ini changes for existing configurations
(via MarvelClient Essentials or Policy)
– Remove CREATE_R85_DATABASES=1
– Remove CREATE_R9_DATABASES=1
– Add CREATE_R10_DATABASES=1
– Add NSF_UpdateODS=1
– Add V10_WORKSPACE_BACKGROUND_MODE=1 → see next slide

Notes 10.x / 11.x – Workspace
• This will be controlled by a notes.ini parameter:
V10_WORKSPACE_BACKGROUND_MODE=0 → default (blue)
V10_WORKSPACE_BACKGROUND_MODE=1 → classic (grey lines)
V10_WORKSPACE_BACKGROUND_MODE=2 → customized
• If you choose for “2”, make sure you have an image resource in your local
bookmark.nsf named “$WorkspaceImage”

Test, pilot, production
• Test
– Try to use virtual machines with snapshots
– Use images, which is very close to your production configuration
▪ Desktops vs. Laptops vs. VDI
▪ ActiveDirectoy → GPOs (Group Policy Objects)
▪ No local admin user rights
▪ Enabled UAC (User Account Control)
▪ Installed AV (Anti-Virus) Software

Test, pilot, production (2)
• Pilot
– Choose the members of the pilot group wisely and let them know
– 20 – 50 users from different departments and with different requirements on IBM Notes
• Production Deployment
– Inform your ServiceDesk and your users before you push the button
– Balance your deployment by upgrading e. g. 100-500 users at the same time – no more

How to run an Upgrade
to Notes V11 Beta-1
in < 5 minutes
Live

Notes on Citrix / VDI

Virtual, Faster, Better – How To Virtualize IBM Notes V10
https://www.slideshare.net/christophadler50/panagenda-webinar-virtual-
faster-better-how-to-virtualize-ibm-notesv10

To roam or not to roam: that is the question
• Given the previous reasons why HCL Notes in virtual environments should
not have any files in the network, the following logical questions arise
– How can personal HCL Notes data folders of end users be maintained locally on the
virtual environment servers?
• Think of having N servers in the farm where personal end user data potentially
needs to be on every single server
• How do end users get their most recent HCL Notes configuration no matter
on which server in the farm they log on?
• How to ensure that disk space used by HCL Notes data folders is cleared
after log off?

To roam or not to roam: that is the question (cont.)
• How to avoid transfers of large HCL Notes data folders per user during log on and
log off which increase wait times and cause network traffic?
• The answer is: Roaming!
– To provide end users with their latest HCL Notes configuration, their data needs to be
roamed onto the server they log on to start their session
– Roaming can be achieved through own procedures (e. g. logon script),
Domino Roaming or third-party solutions

To roam or not to roam: that is the question (cont.)
Log on to
virtual session
Roam
data
Start
HCL Notes
Log on to
HCL Notes
(happens here:
logon script,
3rd-party
solutions)
Roam
data
(or here: MarvelClient
Roaming)
Work with
HCL Notes
Roam
data
(or here:
IBM/HCL
Roaming)
Shutdown
HCL Notes
Log off from
virtual session
Backup
data
Backup
data
(happens here:
MarvelClient
Roaming)
(or here)
Clear data

MarvelClient Roaming
Live

Troubleshooting

Where are the logs?
HCL Notes - Workspace
– Dataworkspacelogs
• Error-log-n.xml
All Java error messages (System.err), stack traces and log messages (Level CONFIG or higher)
• Trace-log-n.xml
All Error-log messages + all active plugin messages
Loglevel:
→ DataWorkspace.configrcpinstall.properties
→ C:Program Files (x86)HCLNotesframeworkrcpeclipsepluginscom.ibm.rcp.base_10.0.0.20190824-
0659rcpinstall.properties
• Default → the last 5 Error-log-n.xml + 10 Trace-log-n.xml will be saved
→ com.ibm.rcp.core.internal.logger.boot.RCPTraceHandler.count=10
→ com.ibm.rcp.core.internal.logger.boot.RCPLogHandler.count=5

Where are the logs? (2)
HCL Notes - IBM_TECHNICAL_SUPPORT
– DataIBM_TECHNICAL_SUPPORT
• console.log & console_<hostname>_<date>@<time>.log
– Incl. MarvelClient log.xml messages
• nsd_W32I_<hostname>_<date>@<time>.log
• core*.dmp
• …
MarvelClient
– %LocalAppData%panagendamarvel
• Logs → the last 5 logs
• Log.xml
• Actions.xml
• Temp*.xml

Performance

Why your Domino servers might have been using 30% more CPU
recently – and how to fix it
• A new feature added in Notes 9.0.1 FP9 (still exists in V10 and V11 Beta-1) called
“Delegated mail files are refreshed automatically“
– Even though the documentation states “Ask your administrator to enable this feature.”, the
functionality is turned on by default
• When an end-user opens another user’s calendar, HCL and IBM Notes will automatically add
some notes.ini entries
DELEGATED_MAIL_FILE1=Christoph Adler_server1/panagenda-demo!!mailcadler.nsf_1/31/2019
DELEGATED_NEWMAIL_SEQNUM1=7
• Impact
HCL and IBM Notes opens many different user mail files upon client startup and regularly thereafter
• Fix
Remove all DELEGATED_* entries in notes.ini
Add DEBUG_DISABLE_AUTO_REFRESH_DELEGATED_INBOX=1 to notes.ini to disable the feature
More information:
https://www.panagenda.com/2019/01/why-your-domino-servers-might-have-been-using-30-more-cpu-recently-and-how-to-fix-it/

Antivirus real-time scanning in Data directory
~55% of the entire data volume in the Data
directory changes on a DAILY basis in most
setups
• 5 - 10% in Dataworkspace (up to 300 files)
• DISABLE Antivirus scanning for at least
*.ns*
*.ndk
*.xml
*.properties
*.jar
in Data Directory.
55%
45%
Data directory:
File changes per day
Once or more Less

What makes the Notes Client START slow
Client crashes/database fixups
• Especially bad for network based data directories clients
Outdated Hardware
• You need >4 GB memory if you want to use Notes AND something else
• Old fixed disks with fragmentation
• Fun Exercise: Buy a Solid State Disk and make Notes start off of that ;-)
Tons of startup programs/scripts launching during the “boot storm”
• If the disk is unable to keep up, it‘s not Notes' fault

What makes the Notes Client START slow (cont.)
Outdated Data directories
• Virtually 100% of all NTFs (Laptops: 90%) can be removed by
• Cleaning up old Notes 4+5+6/7 templates
• using ini:SharedDataDirectory= (also with Single User install!)
Operating System
• Windows 8.1 boots ~20% faster than Windows 7
• Windows 10 ~15% faster than 8.1

ODS of local databases
ODS = On Disk Structure
• ODS 16 = Notes 2
• ODS 43 = Notes 6 & 7
• ODS 51 = Notes 8.5 & 9
• ODS 52 = Notes 9.0.1
• ODS 53 = Notes 10 & 11
The difference between ODS 43 and 53 = up to 80% LESS FILE I/O
This helps with ANY client AND server hardware

ODS of local databases (cont.)
CAUTION: The ODS is not automatically updated when upgrading clients
– ODS 52 requires Notes >= 9.0.1 and CREATE_R9_DATABASES=1 in notes.ini for NEW db’s
– ODS 53 requires Notes >= 10 and CREATE_R10_DATABASES=1 in notes.ini for NEW db's
Good News for Notes (since 8.5.x)
– Use NSF_UpdateODS=1 with CREATE_R9/R10_DATABASES=1 for a one-time upgrade of most local
databases
• NOTE: USE WITH EXTREME CARE IF YOUR DATA DIRECTORIES ARE ON A NETWORK DRIVE!
• NOTE: End users can't access databases during compact OR databases will be ignored if in use
Manually upgrade ODS of local databases/templates
– Run the following command line:
ncompact.exe -C -ODS -* -Client -UpdateIndexes

What makes the Notes Client PERFORM slow
Again
• ODS of local databases
• Hardware
Notes.ini
• EXTMGR_ADDINS = (not just) local virus scanners
• Disable Notes “plugins“ like virus scanners etc.
– They increase Network traffic by up to 200 – 250% and will cause much more CPU load and File I/O
Data directory on the network vs. fixed disk
• Disk performance is the #1 bottleneck
– think SSD vs. normal HDD ; 7.200rpm vs. 5.400rpm ; network drive vs. local fixed disk

Port settings
Port settings are included in the notes.ini
• Type / Active / Encryption / Compression / Timeout
Legacy
• LAN0 / COMx / DisabledPorts → should be removed
Good idea
• Use TCPIP according to the client type
• PORTS = TCPIP
• TCPIP = TCP,0,15,0,,12320, → Desktops
• TCPIP = TCP,0,15,0,,45088, → Laptops
• TCPIP = TCP,0,15,0,,12288, → Citrix / VDI
• Hardware compression (e. g. Riverbed)?
Don‘t forget the Domino servers!

Manage Port Settings
Live

jvm.properties file
To speed up the HCL Notes JVM it’s useful to optimize memory settings
Open %ProgramFiles(x86)%IBMNotesframeworkrcpdeployjvm.properties
– Modify vmarg.Xmx from “-Xmx256m” to “-Xmx1024m”
– Modify vmarg.Xms from “-Xms64m” to “-Xms256m”
– Change vmarg.Xmca=-Xmca512k
The implicit rules
– Do not change vmarg.Xmx to more than -Xmx1024m regardless of how much memory you have
– vmarg.Xms should always be ¼ of what you changed Xmx to
Note
– The above defaults have been changed by IBM/HCL over the last couple of Notes releases.
You might find other values than Xmx256m and Xms64m in your setups.

What makes the Notes Client PERFORM slow (cont.)
IBM Roaming Users
– Try replicating 80MB each time you logon to Notes before use!
Connection documents
– Pointing to outdated server names or IP addresses
– Low priority connection documents, which can cause connectivity failures
Location documents
– IP addresses for mailserver → breaks Policies
– Outdated mail and home server information
– Outdated mail file path
– Catalog server (think db links, doc links, new bookmarks)

Why is the Catalog important?
Ever wondered what exactly happens when you click on db links / doc links / view
links
1. Notes checks whether you already have a workspace icon pointing to the replica ID in the link. If
yes, the topmost icon on the stack decides on which server the db will open
2. If there's no icon on the workspace, Notes asks the catalog server in your currently selected
location document where the db is to be found. So, make sure your catalog is up to date! No
catalog server? Notes will ask your home mail server!
3. Are you replicating your catalog.nsf across all servers? Bad news: If 2.) applies, Notes will open the
database on the server it finds first in the catalog – which is sorted alphabetically. In global
infrastructures that might be a server far far away, e. g. in Asia
Make sure to use an appropriate catalog server in your location documents!

Performance Measuring with Client Clocking
NOTE:
All of this should only be done on “ONE” client – slows down client, creates huge log
files!
Use the following to see and measure client performance
– CLIENT_CLOCK=1
– CONSOLE_LOG_ENABLED=1
– Debug_Outfile=c:debug.txt
– Debug_Console=1
– DEBUG_TCP_ALL=1 (use only if needed)
Fileserver or local disk Performance Debugging is virtually impossible
– If on >= 8.5 try adding DEBUGGINGWCTENABLED=16386 in Notes

Performance Measuring with Client Clocking (cont.)
Client_Clock=1 leads to a debug file of the following format:

OpenNTF – NRPC Parser
Link:
http://www.openntf.org/internal/home.nsf/project.xsp?action=openDocument&name
=Notes%20RPC%20Parser
Last Update:
v1.0.14 – Nov 10, 2016
Developed by:
Jamie Magee, MartinScott Consulting LLC, US
Andrew Magerman, Magerman Collaborative Software GmbH, Switzerland

OpenNTF – NRPC Parser

Performance Tuning – Understanding cache.ndk
We've all been “advised to”/guilty of deleting the cache.ndk in an effort to cure Notes
performance problems/symptoms
– Some companies even delete cache.ndk on a scheduled basis
HOWEVER, let us explain the impact of that on Notes performance ...
– Increases traffic between client and server by 4,000% (forty times more traffic, yes)
(10,000+% if the data directory is on a network drive)
– The more databases end users use, the worse it gets
– There is always a root cause – deleting cache.ndk only fixes symptoms
– Let's look at some detailed traffic analysis for just ONE application (the mail file) of ONE user ...

Performance Tuning – Understanding cache.ndk (cont.)
100 users =
150 MB vs. 3,7 MB
1000 users =
1.5 GB vs. 37 MB
Excluding Fileservers!
After deleting
cache.ndk
Keeping / NOT deleting cache.ndk

Q & A

Thank you!
Christoph Adler
@cadler80
Senior Consultant
panagenda
christoph.adler@panagenda.com

Headquarters, Austria:
panagenda GmbH (Ltd.)
Schreyvogelgasse 3/10
AT 1010 Vienna
Phone: +43 1 89 012 89
Fax: +43 1 89 012 89-15
E-Mail: info@panagenda.com
Headquarters, Germany:
panagenda GmbH (Ltd.)
Lahnstraße 17
DE 64646 Heppenheim
Phone: +49 6252 67 939-00
Fax: +49 6252 67 939-16
USA:
panagenda Inc.
60 State Street, Suite 700
MA 02109 Boston
Phone: +1 617 855 5961
Fax: +1 617 488 2292
Germany:
panagenda Consulting GmbH (Ltd.)
Donnersbergstrasse 1
DE 64646 Heppenheim
Phone: +49 6252 67 939-86
Fax: +49 6252 67 939-16
The Netherlands:
Trust Factory B.V.
11th Floor,
Koningin Julianaplein 10
NL 2595 AA The Hague
Phone: +31 70 80 801 96
E-Mail: info@trust-factory.com
© 2007-2015 panagenda
Make Your Data Work for You

RNUG - Dirty Secrets of the Notes Client

More Related Content

What's hot

Similar to RNUG - Dirty Secrets of the Notes Client

More from Christoph Adler

Recently uploaded

RNUG - Dirty Secrets of the Notes Client