Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

IBM ConnectED 2015 BP110: Mastering Your Logs, Everything You Should Know about Logging in IBM Domino


Published on

Properly logging and monitoring what happens in your Domino environment is critically important for both security and performance. In order to get the most out of your log data when things go wrong, it’s vital to understand its structure, how and what is (or isn’t) logged, and how to search logs effectively. In this in-depth session we will talk about the inner workings of various Domino logging mechanisms by dissecting the structure of log event documents such as Miscellaneous, Replication, Usage Session, User Activity etc. You’ll learn how to deal with verbose logging, retention best practices, monitoring bottlenecks, as well as behind-the-scenes data such as status codes and how to best use them. After this presentation you will walk away with a solid understanding of your log architecture, the means to set up notifications for when things go wrong and faster ways to find what you’re looking for.

Published in: Software

IBM ConnectED 2015 BP110: Mastering Your Logs, Everything You Should Know about Logging in IBM Domino

  1. 1. BP110: Mastering Your Logs Everything You Should Know about Logging in IBM Domino Ben Menesi, Ytria
  2. 2.  Head of Product at Ytria  IBM Notes Administration & Development for various enterprise IBM customers  Certified Advanced Domino Developer & Admin and Security Professional v7+  Speaker at ICS / ESS technology conferences around the globe  IBM Champion for Collaboration Solutions 2014 & 2015 Speaker: Ben Menesi @BenMenesi
  3. 3.  Founder, CEO & CTO at Ytria  15+ years IBM Notes & API development  Speaker at ICS conferences & published author  IBM Design Partner Co-Author Eric Houvenaghel
  4. 4. Domino Server The Admin
  5. 5. 1. • Events take place 2. • Events are logged 3. • Some events captured 4 • Few events addressed 5 • Lots of events missed, angry users on the phone
  6. 6.  Everything about LOG.NSF – Understanding different log events – Extracting additional data – Log Retention, Searching the log and more  User Activity Logging – Understanding what user activities are logged – Utilizing Recorded user activity  Replication Logs – Replication History tips & tricks – Understanding what / where / why – Replication Triangulation Agenda
  7. 7. LOG.NSF What is (or isn’t) logged in log.nsf? How can we extract more data? Best Practices for retention, searching and more
  8. 8.  Almost everything from your Domino server (HTTP Web Server is another topic!) – Generally 90%+ of what’s in your Domino console will be in here • Except for Traveler and some DAOS command logs Log.nsf What is being logged? Log.nsf replica amgr router
  9. 9.  MAIN Log.nsf setting in your server’s Notes.ini file  Logfilename: typically log.nsf  Log_option: 1 = Log to the console, 2 = Force DB fixup when opening log file, 4 = Full document scan  Days: # of days logs will be kept*  Size: Size of log text in event documents*  Days2: Optional # of days to keep activity trends data documents Log.nsf Log file structure LOG=logfilename, log_option, cant_touch_this, days, size, days2
  10. 10.  Understanding your log views Log.nsf Log file structure  SELECT Form=“Events”  SELECT Form=“Mail Routing”  SELECT Form=“Replication”  SELECT Form=“Security”  SELECT Form=“NNTPEvent”  SELECT Form=“Passthru Connection”
  11. 11.  Server tasks simultaneously log into various log.nsf documents  Miscellaneous Events (form=Events) – Size & content controlled by the Log= server notes.ini parameter (later)  Mail Routing Events (form=Mail Routing) – Mainly updated by the SMTP Server and router tasks  Security Events (form=Security) – E.g. ID Vault messages & errors  Replication Events (form=Replication) – Replica & CLREPL tasks Log.nsf Log file structure
  12. 12.  Understanding your log views Log.nsf Log file structure  Select Form=“Session”  Select Form=“Session”  SELECT Form=“Activity”  SELECT Form=“Activity” | Form=“Size”
  13. 13.  User Session Activity (form=Session) – Sessions this server had with users or other servers  Activity Data (form=Activity) – Populated by the nightly Statistics Log task Log.nsf Log file structure
  14. 14.  Server tasks can log events to multiple types of documents! – But no duplicate logs: different information! – Example: Server 1 replicator opens session to Server 2 as per Connection Document settings Log.nsf Log file structure Misc. Replication
  15. 15.  Miscellaneous events: what do we see? – Some high level info (Server, Start & Finish Time) and – UI only shows ONE text type item with limited information Log.nsf 1.) Miscellaneous events
  16. 16.  What the EventsR6 item tells us us – Used to be a Rich Text item Prior to R6 (whew!) – @If(@IsAvailable(Eventlist);EventList;@IsAvailable(Eventslist);Events List;Events) – EventsList item: only contains the Event time & description for each event  What is an event? Log.nsf 1.) Miscellaneous events Time Status Code Severity Type Addin Name Target Server Target DB Target User
  17. 17.  Miscellaneous events – event attributes logged in multi-value Text items Log.nsf 1.) Miscellaneous events EventList Time & Description EventTime EventSeverity EventStatus EventType Event 1 Time Severity HEX Status Type Time & Description Event 2 Time Severity HEX Status Type Time & Description Event 3 Time Severity HEX Status Type Time & Description Event x Time Severity HEX Status Type
  18. 18.  Miscellaneous events – EventSeverity – Severities are recorded via numbers 0 – 5 where • 0: Unknown • 1: Fatal • 2: Failure • 3: Warning High • 4: Warning Low • 5: Normal Log.nsf 1.) Miscellaneous events
  19. 19.  Miscellaneous events – EventType – Multiple type identifiers in use • Best not to mess with this stuff Log.nsf 1.) Miscellaneous events
  20. 20.  Can we get more data? – We can use formulas and the list of Severities, Events & Status Codes to get a better idea of each Misc. log document  Total # of events / document [Eventlist item] – @Elements(Eventlist)  Total # of unique events [EventStatus item] – @Elements(@Unique(EventStatus))  Highest severity / document [EventSeverity item] – _sev:=@If(@Sort(@Unique(@Text(EventSeverity));[Ascending])[1]="0 ";@Sort(@Unique(@Text(EventSeverity));[Ascending])[2];@Sort(@Un ique(@Text(EventSeverity));[Ascending])[1]); Log.nsf 1.) Miscellaneous events
  21. 21.  Customization options (Cont’d)  Once we have the _sev variable, use icons to display (from DDM resources) – @If(_sev="0";"iconNormal";_sev="1";"iconFatal";_sev="2";"iconFailure ";_sev="3";"iconHigh";_sev="4";"iconLow";_sev="5";"iconNormal";"")+" .gif“  # of Fatal severities / document [EventSeverity item] – @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"4":"3":"2":"0 ";"")))  # of Failure severities [EventSeverity item] – @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"4":"3":“1":"0 ";""))) Log.nsf 1.) Miscellaneous events
  22. 22.  # of High Warning severities / document – @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"4":"1":"2":"0 ";"")))  # of Low Warning severities / document – @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"1":"3":"2":"0 ";"")))  # of Normal severities / document – @Elements(@Trim(@Replace(@Text(EventSeverity);"1":"4":"3":"2":"0 ";"")))  # of Unknown severities / document – @Elements(@Trim(@Replace(@Text(EventSeverity);"5":"4":"3":"2":"1 ";""))) Log.nsf 1.) Miscellaneous events
  23. 23.  Customization options: this is what you’ll get – Tip: we can make the Highest and severity columns sortable but careful with your indexes! Log.nsf 1.) Miscellaneous events
  24. 24.  Mail Routing Events – Same Structure as Miscellaneous events – Incrementally fills documents (considering 40KB / Log= limit) Log.nsf 2.) Mail Routing Events Time EventList EventStatus EventSeverity
  25. 25.  Is there any additional data we can mine? – Use same custom views (Severities, unique events, etc…) • Tip: Messages can be traced based on the last 8 characters of their Universal ID • Extract list of Messages transferred per Mail Routing Log, or their (unique) numbers. Log.nsf 2.) Mail Routing Events
  26. 26.  Unique # of Messages transferred [EventList item] – Could be used as a sortable column formula • @Elements(@Unique(@Explode(@Implode(@Trim(@Word(@Replac eSubstring(EventList;"Router: Message ":"Router: No messages":" transferred to ":"NOT transferred ":" delivered to ";"§§§":"§§§":"§§§":"§§§":"§§§");"§§§";2));", ");", "))) Log.nsf 2.) Mail Routing Events
  27. 27.  List of unique message codes [EventList item] – Could be used as a categorized column formula – help finding emails • @Unique(@Explode(@Implode(@Trim(@Word(@ReplaceSubstring( EventList;"Router: Message ":"Router: No messages":" transferred to ":"NOT transferred ":" delivered to ";"§§§":"§§§":"§§§":"§§§":"§§§");"§§§";2));", ");", ")) Log.nsf 2.) Mail Routing Events
  28. 28.  Security Event documents – Same Structure as Miscellaneous events – Incrementally fills documents (considering 40KB / Log= limit) – Typically you won’t even need a special view to understand unless you have a very high volume. Log.nsf 3.) Security Events
  29. 29.  Replication Events <> Miscellaneous events – Important: Logs documents PER SESSION – Only Replication Sessions where the current server did the work! • Search all participating server log files to get the big picture! – What is stored in those replication events? Log.nsf 4.) Replication Events InitiatedBy Server SourceServer Body BytesIn BytesOut Pathname EventList EventSeverity EventStatus EventType
  30. 30.  What additional data can we extract and use? – EventList & EventSeverity items ONLY created on error – # of entries in the Pathname item tells us how many events – Body item contains: Access, Additions, Updates, Deletes, Kbytes sent & received for each DB • Body item is NOT multi-value (can’t really work with formulas) • Body item is NOT part of the summary if more than 1 event is logged = can NOT be displayed in views • Cluster Replication events are NOT properly logged unless you use the RTR_Logging parameter • Log.nsf 4.) Replication Events
  31. 31.  Interesting stuff: even when DB1 only has to SEND data to DB2, we can see data being received by DB1 – BytesIN & BytesOUT: NOT reliable: Log.nsf 4.) Replication Events
  32. 32.  Creating a custom replication view  Replicator / Cluster Replicator? [Initiatedby item]  Number of events [Pathname item] – @Elements(Pathname)  Any Errors? [EventSeverity item] – @If(@Elements(EventSeverity)>0;150;"")  Direction [Body item] – _bl1:=@Contains(Body;"PULL"); – _bl2:=@Contains(Body;"PUSH"); – @If(_bl1=1&_bl2=1;"PULL- PUSH";_bl1=1&_bl2=0;"PULL";_bl2=1&_bl1=0;"PUSH";"-") Log.nsf 4.) Replication Events
  33. 33.  Creating a custom replication view (Cont’d) Log.nsf 4.) Replication Events
  34. 34.  Usage Session logs – what do we (not) see? – IMPORTANT: Logging on a /USER/SESSION base! – Contains a TON of raw data Log.nsf 5.) Usage Session Documents
  35. 35.  What is an event? Log.nsf 5.) Usage Session Documents Database Reads Writes Transactions Bytes Read Bytes Written DB Open time UserName Pathname Reads Writes Transactions SentFromServer SentToServer Body
  36. 36.  What additional data can we extract and use? – PathName item (multi-value) contains list of databases accessed • @Elements(PathName) gives us the # of DBs accessed for each session – Rearrange columns in the Usage By User view Log.nsf 5.) Usage Session Documents
  37. 37.  Searching your log file is painful – Reason: you’ll be presented with any matching documents but you won’t be able to read between the lines – Looks familiar? – … and the sad part: 6 results would actually make us happy (RIGHT?) Log.nsf How to search?
  38. 38.  Option #1: search using the Admin client – TONS of parameters to specify – slow setup. Better for periodical search requirements Log.nsf How to search?
  39. 39.  Option #1: search using the Admin client (Cont’d) – Tip: do NOT use the Event Type tab! • You can miss results because events are logged across types Log.nsf How to search?
  40. 40.  Option #1: search using the Admin client (Cont’d) – Tip: You can save queries in domadmin.nsf to reuse later – BEWARE: Search results populate new documents! • Avg. 180 events / document: single search with 1800 matches creates 10 documents • Results can not be analyzed nor copied and are stored across multiple documents – Conclusion: Useful for very specific queries that you may need periodically Log.nsf How to search?
  41. 41.  Option #2: A quicker DIY solution (less customizable) – Advantages: reads matches from multiple log documents and saves & displays them in one text file – Domino-server-log Log.nsf How to search?
  42. 42.  Option #3: Pro tip for data-range search (Kudos to Kim Greene) – Select log documents and Actions Forward • Aggregates EventList item contents in one document • Use CTRL+F to search Log.nsf How to search?
  43. 43.  Option #4 Great open source log parser via OpenNTF from Jakob Majkilde – Installed on server, access to Console Logs, Trace Logs, Log.nsf and more with a nice Xpages interface – File%20Reader Log.nsf How to search?
  44. 44.  Option #5+ 3rd Party tools – Use a tool like Ytria consoleEZ Log.nsf How to search?
  45. 45.  Log deletions managed by parameter defined in log= server notes.ini parameter – Deletions made at once, all entries older than 7 days removed – Add to that: Deletion Stub Purge interval • Default 90 days could result in =< 120 days old deletion stubs • Deletions happen every 1/3rd of the time defined here Log.nsf Log file retention
  46. 46.  Known problem in previous Domino versions: log.nsf continues to grow and ignores log= settings – Workaround: use “Remove documents not modified in the last (days) • Beware: this field also defines Deletion Stub Purging time • Tip: Deleted documents are removed WITHOUT creation of deletion stubs Log.nsf Log file retention
  47. 47.  Recommendation – Use the “Remove documents note modified in the last (days)” setting – Use console logs to keep old log data Log.nsf Log file retention
  48. 48.  Log_AgentManager – 0 for no logging, 1 for partial & successful agent execution events, 2 for successful events only  Log_Replication – 1 for logging when DB replicates, 2 for summary info about each DB, 3 for detailed info about each replicated note • Great for debugging!  RTR_Logging – 1 (Default), 4 for Logging replications (attempted & performed) • Cluster replication will be logged Log.nsf A few notes.ini parameters in the back pocket
  49. 49.  Mail_Log_To_MiscEvents – 0/1 whether to log mail events into Misc. event documents • Tip: MailLogToEventsOnly=1 to NOT show router messages in the Domino Console  SMTPClientDebug=1 – Use temporarily to debug outbound mail problems • Tip: don’t get confused by client, this one’s for your server!  HTTPLogUnauthorized=1 – Logs 401 HTTP errors in server console AND Misc. log events • HTTP Users attempting to access resources that aren’t available and failed user authentication requests • This is GREAT: brings a bit of web server logs into your log.nsf Log.nsf A few notes.ini parameters in the back pocket
  50. 50.  Log_Console=2 – Logs all console commands even if prefixed with ! (By default !sh task won’t be logged!)  LOG_DisableTXNLogging=1 – 0/1 Take your Log.nsf out of DBs to be transaction logged • Also clubusy.nsf and!  No_Force_Activity_Logging – 0/1 Controls whether the statlog task automatically enables activity recording for all DBs (Default = 0) • Tip even if disabled activity is being recorded in the Log.nsf Usage views Log.nsf A few notes.ini parameters in the back pocket
  51. 51. USAGE ACTIVITY What are our most used databases? How do we know what a given user has been up to? How do we get user activity for multiple databases?
  52. 52.  User Activity can be a *VERY* rich source of data – Enable in the database properties window • Tip: No_Force_Activity_Logging=0 (Default value) allows on ALL dbs • Tip: Set to confidential to prevent users with lower than Designer access from accessing it Database User Activity Recorded User Activity
  53. 53.  Did you upgrade your database On Disk Structures? – I’m not proud of it – Tip: use Compact –REPLICA for upgrading system database ODS’s without downtime! Database User Activity Before we go any further
  54. 54.  Prior to ODS 48 – Entry size allocated 44 bytes, Object size 61600 bytes (max 1400 entries)  ODS 48+ (undocumented improvement) – Entry size allocated 92 bytes, Object size 128800 bytes (max 1400 entries)  Curious: only two additional counters added, yet ODS 48+ entry is over 2x the size of those prior to ODS48: Why? Database User Activity User Activity – What is really captured? Date & Time Reads Adds Updates Deletes User Date & Time Reads Writes User
  55. 55.  Does Notes only capture as much as it displays? – Nope Database User Activity User Activity – What is really captured? Reads Adds Updates Deletes Data Non Data Σ Data Non Data Σ Data Non Data Σ Data Non Data Σ
  56. 56.  Differentiating between Data (as in: Document class notes) and Non-Data (as in: everything else such as Designs, ACLs, etc…) can help us look for more specific scenarios: – Databases where no DOCUMENT class notes were read for some time – Databases where no DOCUMENT class notes were added or updated since awhile – Users who’ve deleted Designs or ACLs – And more. Database User Activity User Activity – Data & Non-Data
  57. 57.  Makes no difference between users and servers – Workaround: use a tool to parse the data and input exceptions  Only maintains 1400 entries – Wait, this doesn’t need to be a problem!  Still answers questions like “What databases aren’t used on a regular basis?” – Databases that contain 1400 entries are properly used apps • @Date(FirstEntry) - @Date(LastEntry) tells us how frequently used – Databases with lower than 1400 entries indicate they aren’t used that much Database User Activity User Activity - Downside
  58. 58.  How do we get the user activity for multiple databases – Third party tools like – …or Ytria’s databaseEZ – …or we can implement it ourselves using the NotesUserActivity class: • Uses the W32_NSFDbGetUserActivity API call (restrictive: no Data vs. Non-Data) 68525688d00708397?OpenDocument Database User Activity User Activity – How to get it?
  59. 59. REPLICATION LOGS How is replication logged? What can we read out? What to watch out for?
  60. 60.  Set via Log_Replication=value – 0 Do not log replication events – 1 Log that DB is replicating (Default value) – 2 Log summary info about each DB – 3 Log info about each replicated document (both design and doc class) – 4 Log info about each replicated field  Note: only impacts logging of replication events performed by the current server! Replication Logs Replication verbosity logs
  61. 61.  Keeps track of what the current database replicated with – Tip1: Local <> Server replication leaves NO TRACE in server replica’s replication history! – Tip2: When there’s nothing to replicate, no replication history entry created Replication Logs Replication History
  62. 62.  Feature aiming to optimize performance introduced in Domino 7 – Result: Replication History Entries indicating replication between A & C which never happened Replication Logs Replication Triangulation A B C A – B A – B B – C B – C A – C
  63. 63.  Idea: to prevent complete recalculation when replicating with an unknown server – Results show 400% CPU increase in some cases (1700 servers) – Disable (server side): • NSF_REPLHIST_NO_TRI=1 • REPL_NO_WS_TRI_HIST=1 • REPL_NO_REMOTE_TRI_HIST=1 – Gotcha: need to clear replication history entries for all impacted DBs! – Disable (client side): • NSF_REPLHIST_NO_TRI=1 [No triangulated entries read] • REPL_NO_WS_TRI_HIST=1 [No triangulated entries written] Replication Logs Replication Triangulation
  64. 64.  Purge Interval Replication Control – New(ish) awesome option for preventing old documents from coming back (Introduced in Domino 8.5.3) • Set via Replication Options > Space Savers – DEBUG_REPL_PIRC=1 gives you date and summary info about documents not allowed to replicate – DEBUG_REPL_PIRC=2 or greater provides more details about documents blocked by PIRC Replication Logs PIRC logging
  65. 65.  Please fill out your evaluation forms – Your feedback is very important!  Feel free to get in touch! – Contact - Ben Menesi Thanks Thank you for attending! @BenMenesi
  66. 66. Engage Online  SocialBiz User Group – Join the epicenter of Notes and Collaboration user groups  Social Business Insights blog – Read and engage with our bloggers  Follow us on Twitter – @IBMConnect and @IBMSocialBiz  LinkedIn – Participate in the IBM Social Business group on LinkedIn  Facebook – Like IBM Social Business on Facebook
  67. 67. Copyright © 2015 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM. U.S. Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM. Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. THIS DOCUMENT IS DISTRIBUTED "AS IS" WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IN NO EVENT SHALL IBM BE LIABLE FOR ANY DAMAGE ARISING FROM THE USE OF THIS INFORMATION, INCLUDING BUT NOT LIMITED TO, LOSS OF DATA, BUSINESS INTERRUPTION, LOSS OF PROFIT OR LOSS OF OPPORTUNITY. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided. Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice. Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation. It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actions the customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third- party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right. IBM, the IBM logo,, BrassRing®, Connections™, Domino®, Global Business Services®, Global Technology Services®, SmartCloud®, Social Business®, Kenexa®, Notes®, PartnerWorld®, Prove It!®, PureSystems®, Sametime®, Verse™, Watson™, WebSphere®, Worklight®, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: Notices and Disclaimers