Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on

  • Be the first to comment


  1. 1. Compact, Compress, De-Duplicate Ulrich Krause ILUG, 10. – 12. November 2010, Belfast, Ireland
  2. 2. About: Ulrich Krause Lotus Notes / Domino Administrator & Developer since 1993 OpenNTF project chef – !!HELP!! – Cebulon – TriggerHappy German Notes Forum ( Blog
  3. 3. The situation More and bigger documents and attachments More expensive storage is needed The average size of an application increases Duplicate attachments ( n times ) Duration of standard tasks increases More time for Backup Network traffic increases I / O and performance issues
  4. 4. The aim Leverage existing storage Reduce size of applications Avoid attachment redundancies Speed-up standard tasks Reduce backup time Reduce network traffic Reduce disc I / O
  5. 5. Application model ( simplified ) Design Documents – Text – Attachments – Embedded objects Design Text Attachments
  6. 6. Compact Secrets DEBUG_ENABLE_COMPACT_8_5=1 – W nn - Only compact databases which haven't been compacted successfully in the last nn days. – w - Exclude system databases (e.g., log.nsf, names.nsf). – X nn - Limit each database compaction to no more than nn minutes. (Applies to in-place compaction only) – x nn - Limit total compaction time to nn minutes. – # nn - Execute up to nn compactions in parallel using different threads. • Don‟t use more than 8 threads in parallel – ODS performs a copy-style compact only on databases with an ODS smaller than the ODS provided with this parameter.
  7. 7. Compact Secrets (Sample) Load compact -# 8 mail Compacts all applications in the mail subdir in 8 parallel Threads
  8. 8. Compact via application properties Compacting a database does not necessarily reduce its size If the Notes database requires structural changes (typically a change of ODS [on-disk structure]). then a copy-style compaction will be performed (equivalent to -L). If the Notes database is logged (Transactional Logging enabled), then a simple in-place compaction will be performed (equivalent to -b). If the Notes database is not logged (Transactional Logging disabled), an in-place with file size reduction compaction will be performed (equivalent to -B).
  9. 9. Copy Style Compaction The compact -C (copy-style compaction) (1) creates a new empty application container on the same server with the same replica ID and a .tmp suffix, (2) copies all of the documents from the original application to the new application container. After all documents have been populated in the new application, (3) the old .nsf application is deleted and the (4) .tmp file is renamed to the original filename.
  10. 10. Defrag File fragmentation saps Domino server I/O performance Even a slightly fragmented file can cause a 50% decrease in non-cached read performance Load compact –C does not allocate space needed before compacting the application Applications are fragmented during copy-style compaction Full Text Indexes are fragmented too Even NLO files in DAOS repository are fragmented over the time Use a tool to defrag your files speed up your Domino server !
  11. 11. Defrag DefragNSF • Adam Osborne • Domino Defrag • Andrew Luder, OpenNTF • ojectLookup/DominoDefrag Both products use the Microsoft defragmentation API
  12. 12. Defrag Sample: Lotus Software KnowledgeBase ( 5 * compact –c ) Before – 68 fragments – 7550 ms After – 1 fragments – 125 ms
  13. 13. Defrag Domino Defrag uses IBM Lotus Notes C API Open Source FREE Well documented 32/64 Bit support New in Version 3: Free Space Defragmentation – Uses jkdefragcmd.exe ( There is also a professional support plan available
  14. 14. Design Note Compression
  15. 15. Design Note Compression New feature as of Domino 8 Needs ODS 48 or above – Create_R8_Databases=1 – or Create_R85_Databases=1 – load compact -c – load compact -c –n Check “Compress database design” Reduces size of design by 55 – 60 % Reduces disc I / O
  16. 16. Design Note Compression Design Text Attachments Design Text Attachments
  17. 17. Document Body Compression
  18. 18. Document Body Compression New feature as of Domino 8.0.1 Reduces size of Rich Text up to 60% Needs ODS 48 or above load compact -c –v Check „Compress document data“
  19. 19. Document Body Compression Design Text Attachments Design Text Attachments
  20. 20. LZ1 Attachment Compression
  21. 21. LZ1 Attachment Compression As of release 6 Replaces Huffmann algorithm NOT active when creating a new applications. Recompress using • load compact -c –ZU • In 6.5.6 also set DEBUG_ENABLE_LZ1_REPAIR=1 To avoid problems with JAVA applets in <= 8.5 set • DEBUG_ENABLE_LZ1_HOST_TYPES=0 • fixed in 8.5.1
  22. 22. LZ1 Attachment Compression There have been problems reported that the fulltext search does not work as expected using LZ1 compression. To revert back to Huffmann use – Load compact –c –ZD – In 6.5.6 also set DEBUG_ENABLE_LZ1_REPAIR=1
  23. 23. LZ1 Attachment Compression Design Text Attachments Design Text Attachments
  24. 24. DAOS Estimator
  25. 25. DAOS Estimator Free tool to plan your DAOS rollout • Download: http://www-
  26. 26. DAOS Estimator (notes.ini) DAOS Estimator by default has a large gap in attachment size evaluation between 64k and 1 MB – DAOSEST_BUCKETS=16,64,128,256,512,768,1024,2048,3072,4096 – The above line sets to: – 16K, 64K, 128K, 256K, 512K, 768K, 1MB, 2MB, 3MB and 4MB Source: Lotus Software Knowledge Base • http://www-
  27. 27. LZ1 != HUFFMAN Db1 – LZ1 enabled Db2 – LZ1 disabled  Result is 2 .NLO Files !!
  28. 28. D A O S
  29. 29. DAOS Domino Attachment Object Service Attachments are no longer stored in application Attachments are stored as NLO (Notes Large Object ) Document contains only a ticket Separates static and dynamic data Only one copy of a bit-like attachment Transparent to users and 3rd-Party Products
  30. 30. Prerequisites Domino 8.5 ( ODS 51) SHARED MAIL OFF Transaction protocol Create DAOS Repository Activate DAOS on server Activate DAOS on application
  31. 31. Switch OFF Shared Mail Serverdocument – Shared Mail Field Shared Mail = None „Load object unlink shared.nsf“
  32. 32. Transaction Protocol Server document – Transactional Logging Required for DAOS – no discussion!! Use a separate disk drive. Create log file folder Choose method • Circular » 4GB limit • Archive » No limit, backup will delete logs • Linear » Like circular, NO size limit CREATE_R85_LOG=1 Restart server
  33. 33. DAOS Repository Use separate disk drive Can be SAN, NAS Each server has it„s own repository – Max.1.000 subdirectories – Max 40.000 .NLO / subdirectory – = 40.000.000 unique .NLOs – Daos.cfg ( in Domino Data Directory) – Daoscat.nsf .NLO files are encrypted with – DAOS_ENCRYPT_NLO=0 Include repository in backup!!
  34. 34. DAOS Repository How do you change the number of DAOS NLO's per directory? DAOS_MAX_FILES_PER_SUBCONTAINER=40000 Requires a server restart Will only affect the current DAOS subdirectory (older subdirectories are not changed). This will also affect the maximum number of files in the DAOS store which is currently this values multiplied by 1000 (this value is not configurable currently). • 1.000 * DAOS_MAX_FILES_PER_SUBCONTAINER = MAX_NLO
  35. 35. Activate DAOS on server Goto Serverdocument – DAOS Set Minimum size of an object • recommended 64kB (= DEFAULT in 8.5.1 ) • Evaluate with DAOS Estimator Set DAOS base path • Put is outside the data path! Set Delete objects in repository • Depends on your backup strategy • 0 Days = immediate delete after the refcount = 0 • Manually from console: TELL DAOSMGR PRUNE 0  Restart the server  show server
  36. 36. Activate DAOS on applications Application  properties  „propeller hat“ for single application Use Admin client for multiple applications Valid for new attachments only! To move existing attachments into the DAOS repository – Load compact –C –DAOS ON <PathNSFs>
  37. 37. Bug Alert !!! load compact -c –DAOS ON mail (w/o trailing backslash) enables DAOS on ALL applications, not only on those in mail subdirectory !! Documented Lotus Software Knowledge Base #1410926 SPR AHOE7XHM9J (fixed in 8.5.2)
  38. 38. The Result Design Text Tickets Design Text Attachments DAOS Repository
  39. 39. Transaction Protocol Trap load compact –c –DAOS ON NO attachment in DAOS repository tell daosmgr dbsummary count = 0 load compact -C -T -DAOS on
  40. 40. DAOS and replication DAOS is a server feature! On a client replica, all attachments are stored in the application. Replicas on a NON-DAOS server store attachments in the application Each server in a cluster has it„s own DAOS repository.
  41. 41. DAOS and replication (cont.)  Client  Server (as of 8.5.1) – „known” attachments are no longer transferred to server = „smart replication“ • Forward mail • Reply with history Server  Server (as of 8.5.1) – “known” attachments are no longer replicated to target server 8 new statistics – show stat DAOS
  42. 42. DAOS and replication (cont.) Domino 8.5 Domino 8.5.1 WARNING!!: ADMINP_ACCELERATED_REPLICA_OVERRIDE=4 !!
  43. 43. DAOS and replication (last slide )
  44. 44. DAOS and Remove RM_NO_LOG_OBJECTS_IN_MAILBOX from notes.ini. Enable DAOS on all files Enable DAOS on all mail files Increases performance
  45. 45. DAOS and Encryption Increases number of NLO files! – User A writes an encrypted mail with attachment to User B – User B writes an encrypted mail with same attachment to User A – Results in 4 NLO Files in repository Does not save space but separates static from dynamic data
  46. 46. DAOS and Quotas Technote: 1405456 Depends on the quota enforcement setting that is in place Check space used in file when adding a note – Acts on the physical size of the application (without attachments) – Possibly a problem when switching DAOS off!! • 64GB limit Check file size when extending the file Check file size when adding a note – Both acts on the logical size of the application (with attachments)
  47. 47. Keep daoscat.nsf in sync DON„T use DDM !! Use program document Tell daosmgr resync SPR# DROO7ZNPZ8 Added a new DAOSMGR resync switch to perform a quick resync. This new switch updates the DIT and DOI but does not run through the applications or updates the NLO refcounts. ( as of 8.5.1 IF1 )
  48. 48. „Needs Resync“ DDM points to the first application that causes the DAOS catalog to go into „Needs Resync“ state. If you cannot find a cause for the issue set debug_threadid=1 console_log_enabled=1 daos_logging=<path/daos.txt>STATE_CHANGE Many improvements in 8.5.2 that reduces the number of state changes (SPR# RCOR83NLQK)
  49. 49. DAOS resync time window feature The following two notes.ini parameters control when resync will be allowed to run – DAOS_RESYNC_START_TIME=12:00:00 AM – DAOS_RESYNC_STOP_TIME=04:00:00 AM Note that these parameters do not launch resync automatically; they just control whether or not resync may run at a given time, and also tell resync when to stop. A program document could be created to launch resync just after midnight. If the catalog is SYNCHRONIZED, then the resync will not take place. However, if a resync is needed, then the resync process will begin. When the end time is reached, the resync threads will finish up the databases they are processing and exit.
  50. 50. DAOS OFF What happens when DAOS is disabled / switched OFF? In application properties – No documents with attachments in the application – Application has documents with attachments • Status in Admin Client changes to „Read Only“ • DAOS still works for already stored attachments but no new attachments are written to the repository for the specific application. In server document – Same as 1
  51. 51. DAOS and missing NLO NLO has been deleted on the OS level TELL DAOSMGR LISTNLO MISSING <application.nsf> Output to <datadir> by default
  52. 52. Daosmgr in Domino 8.5.2 daosmgr options have been enhanced in Domino 8.5.2 You can now do a search for missing NLO on all applications in a subdirectory – tell daosmgr listnlo missing mail
  53. 53. DAOS and missing application NSF has been deleted on the OS level  Message on server startup • Informational - The DAOS catalog is not synchronized. Deletions will be postponed. Please run 'tell daosmgr resync' at the next convenient opportunity to re-synchronize. Has the application been deleted by accident? – NO • TELL DAOSMGR RESYNC oder RESYNC FORCE – YES • Restore from backup • Create a new replica from a cluster mate  Same behaviour when moving an application (on OS level)
  54. 54. Can we save another 50% If we disable encryption, then we could put NLO for all servers into a single repository … Yes, in theory It„s only experimental Does not work because each server has its own daoscat.nsf. Would cause problems with pruning
  55. 55. DAOS and Nagios ( Statistics ) Catalog.Status Repository.Size  Plugin
  56. 56. DAOS amd Statistics 8 new statistics (8.5.1) – show stat DAOS 8.5.1 Debug_NSF_Show_Allstats=1
  57. 57. DAOS and Statistics Windows: @echo off setlocal for /f "tokens=*" %%a in ('dir/w/s %1 ^|findstr "Datei(en)"') do ( set info=%%a ) echo %info% Linux: #!/bin/sh REPOSITORY=/local/daos OUTFILE=/local/daos.txt fCount=`find $REPOSITORY -type f | wc –l` fSize=`du -ksb $REPOSITORY` IFS=”/” array=($fSize) fDate=$(date +”%d-%m-%Y”) echo $fDate / $fCount / ${array[0]}>> $OUTFILE
  58. 58. Use XML
  59. 59. DAOS and daos.cfg Contains information on file count and file size of .NLO files Created on server startup when it has been deleted Updated on server shutdown Do not backup or restore!!
  60. 60. DAOS and daoscat.nsf Contains index and refcount of NLO Contains index of all applications using DAOS Does NOT contain any view Re-Created on re-sync Created on server startup when it has been deleted Do not backup or restore!!
  61. 61. Admin Client ( Files tab ) New columns regarding DAOS – Logical size – Physical size – DAOS status – DAOS count – DAOS size New columns regarding compression
  62. 62. Backup & Restore No change in backing up NSF files. Additional backup job for NLO files NLO files do not require any special agent for backup IMPORTANT: the backup sequence – server is up during backup • NSF before NLO – server is down during backup • it‟s your choice IBM Technote # 1358548 - DAOS Backup and Restore
  63. 63. Resources DAOS Wiki • http://www- at=null&tag=DAOS DAOS Backup and Restore • DAOS Estimator • DAOS Technical Overview • eeba5/b6523e7f4b15585b862575b5004d3488/$FILE/DAOSTechnicalOvervi ew_NELotus.pdf
  64. 64. This is it! Contact me at: Thank you dank u wel merci Danke