Your SlideShare is downloading. ×
Ccd
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Ccd

1,400
views

Published on


0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,400
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
40
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Compact, Compress, De-Duplicate Ulrich Krause ILUG, 10. – 12. November 2010, Belfast, Ireland
  • 2. www.eknori.de About: Ulrich Krause Lotus Notes / Domino Administrator & Developer since 1993 OpenNTF project chef – !!HELP!! – Cebulon – TriggerHappy German Notes Forum (http://www.atnotes.de) Blog http://www.eknori.de
  • 3. www.eknori.de The situation More and bigger documents and attachments More expensive storage is needed The average size of an application increases Duplicate attachments ( n times ) Duration of standard tasks increases More time for Backup Network traffic increases I / O and performance issues
  • 4. www.eknori.de The aim Leverage existing storage Reduce size of applications Avoid attachment redundancies Speed-up standard tasks Reduce backup time Reduce network traffic Reduce disc I / O
  • 5. www.eknori.de Application model ( simplified ) Design Documents – Text – Attachments – Embedded objects Design Text Attachments
  • 6. www.eknori.de Compact Secrets DEBUG_ENABLE_COMPACT_8_5=1 – W nn - Only compact databases which haven't been compacted successfully in the last nn days. – w - Exclude system databases (e.g., log.nsf, names.nsf). – X nn - Limit each database compaction to no more than nn minutes. (Applies to in-place compaction only) – x nn - Limit total compaction time to nn minutes. – # nn - Execute up to nn compactions in parallel using different threads. • Don‟t use more than 8 threads in parallel – ODS performs a copy-style compact only on databases with an ODS smaller than the ODS provided with this parameter.
  • 7. www.eknori.de Compact Secrets (Sample) Load compact -# 8 mail Compacts all applications in the mail subdir in 8 parallel Threads
  • 8. www.eknori.de Compact via application properties Compacting a database does not necessarily reduce its size If the Notes database requires structural changes (typically a change of ODS [on-disk structure]). then a copy-style compaction will be performed (equivalent to -L). If the Notes database is logged (Transactional Logging enabled), then a simple in-place compaction will be performed (equivalent to -b). If the Notes database is not logged (Transactional Logging disabled), an in-place with file size reduction compaction will be performed (equivalent to -B).
  • 9. www.eknori.de Copy Style Compaction The compact -C (copy-style compaction) (1) creates a new empty application container on the same server with the same replica ID and a .tmp suffix, (2) copies all of the documents from the original application to the new application container. After all documents have been populated in the new application, (3) the old .nsf application is deleted and the (4) .tmp file is renamed to the original filename.
  • 10. www.eknori.de Defrag File fragmentation saps Domino server I/O performance Even a slightly fragmented file can cause a 50% decrease in non-cached read performance Load compact –C does not allocate space needed before compacting the application Applications are fragmented during copy-style compaction Full Text Indexes are fragmented too Even NLO files in DAOS repository are fragmented over the time Use a tool to defrag your files speed up your Domino server !
  • 11. www.eknori.de Defrag DefragNSF • Adam Osborne • http://www.preemptive.com.au/defrag Domino Defrag • Andrew Luder, OpenNTF • http://www.openntf.org/Projects/pmt.nsf/Pr ojectLookup/DominoDefrag Both products use the Microsoft defragmentation API
  • 12. www.eknori.de Defrag Sample: Lotus Software KnowledgeBase ( 5 * compact –c ) Before – 68 fragments – 7550 ms After – 1 fragments – 125 ms
  • 13. www.eknori.de Defrag Domino Defrag uses IBM Lotus Notes C API Open Source FREE Well documented 32/64 Bit support New in Version 3: Free Space Defragmentation – Uses jkdefragcmd.exe (http://kessels.com/jkdefrag/) There is also a professional support plan available
  • 14. www.eknori.de Design Note Compression
  • 15. www.eknori.de Design Note Compression New feature as of Domino 8 Needs ODS 48 or above – Create_R8_Databases=1 – or Create_R85_Databases=1 – load compact -c – load compact -c –n Check “Compress database design” Reduces size of design by 55 – 60 % Reduces disc I / O
  • 16. www.eknori.de Design Note Compression Design Text Attachments Design Text Attachments
  • 17. www.eknori.de Document Body Compression
  • 18. www.eknori.de Document Body Compression New feature as of Domino 8.0.1 Reduces size of Rich Text up to 60% Needs ODS 48 or above load compact -c –v Check „Compress document data“
  • 19. www.eknori.de Document Body Compression Design Text Attachments Design Text Attachments
  • 20. www.eknori.de LZ1 Attachment Compression
  • 21. www.eknori.de LZ1 Attachment Compression As of release 6 Replaces Huffmann algorithm NOT active when creating a new applications. Recompress using • load compact -c –ZU • In 6.5.6 also set DEBUG_ENABLE_LZ1_REPAIR=1 To avoid problems with JAVA applets in <= 8.5 set • DEBUG_ENABLE_LZ1_HOST_TYPES=0 • fixed in 8.5.1
  • 22. www.eknori.de LZ1 Attachment Compression There have been problems reported that the fulltext search does not work as expected using LZ1 compression. To revert back to Huffmann use – Load compact –c –ZD – In 6.5.6 also set DEBUG_ENABLE_LZ1_REPAIR=1
  • 23. www.eknori.de LZ1 Attachment Compression Design Text Attachments Design Text Attachments
  • 24. www.eknori.de DAOS Estimator
  • 25. www.eknori.de DAOS Estimator Free tool to plan your DAOS rollout • Download: http://www- 01.ibm.com/support/docview.wss?rs=463&uid=swg24021920
  • 26. www.eknori.de DAOS Estimator (notes.ini) DAOS Estimator by default has a large gap in attachment size evaluation between 64k and 1 MB – DAOSEST_BUCKETS=16,64,128,256,512,768,1024,2048,3072,4096 – The above line sets to: – 16K, 64K, 128K, 256K, 512K, 768K, 1MB, 2MB, 3MB and 4MB Source: Lotus Software Knowledge Base • http://www- 01.ibm.com/support/docview.wss?rs=0&q1=1418102&uid=swg21418102
  • 27. www.eknori.de LZ1 != HUFFMAN Db1 – LZ1 enabled Db2 – LZ1 disabled  Result is 2 .NLO Files !!
  • 28. www.eknori.de D A O S
  • 29. www.eknori.de DAOS Domino Attachment Object Service Attachments are no longer stored in application Attachments are stored as NLO (Notes Large Object ) Document contains only a ticket Separates static and dynamic data Only one copy of a bit-like attachment Transparent to users and 3rd-Party Products
  • 30. www.eknori.de Prerequisites Domino 8.5 ( ODS 51) SHARED MAIL OFF Transaction protocol Create DAOS Repository Activate DAOS on server Activate DAOS on application
  • 31. www.eknori.de Switch OFF Shared Mail Serverdocument – Shared Mail Field Shared Mail = None „Load object unlink shared.nsf“
  • 32. www.eknori.de Transaction Protocol Server document – Transactional Logging Required for DAOS – no discussion!! Use a separate disk drive. Create log file folder Choose method • Circular » 4GB limit • Archive » No limit, backup will delete logs • Linear » Like circular, NO size limit CREATE_R85_LOG=1 Restart server
  • 33. www.eknori.de DAOS Repository Use separate disk drive Can be SAN, NAS Each server has it„s own repository – Max.1.000 subdirectories – Max 40.000 .NLO / subdirectory – = 40.000.000 unique .NLOs – Daos.cfg ( in Domino Data Directory) – Daoscat.nsf .NLO files are encrypted with server.id – DAOS_ENCRYPT_NLO=0 Include repository in backup!!
  • 34. www.eknori.de DAOS Repository How do you change the number of DAOS NLO's per directory? DAOS_MAX_FILES_PER_SUBCONTAINER=40000 Requires a server restart Will only affect the current DAOS subdirectory (older subdirectories are not changed). This will also affect the maximum number of files in the DAOS store which is currently this values multiplied by 1000 (this value is not configurable currently). • 1.000 * DAOS_MAX_FILES_PER_SUBCONTAINER = MAX_NLO
  • 35. www.eknori.de Activate DAOS on server Goto Serverdocument – DAOS Set Minimum size of an object • recommended 64kB (= DEFAULT in 8.5.1 ) • Evaluate with DAOS Estimator Set DAOS base path • Put is outside the data path! Set Delete objects in repository • Depends on your backup strategy • 0 Days = immediate delete after the refcount = 0 • Manually from console: TELL DAOSMGR PRUNE 0  Restart the server  show server
  • 36. www.eknori.de Activate DAOS on applications Application  properties  „propeller hat“ for single application Use Admin client for multiple applications Valid for new attachments only! To move existing attachments into the DAOS repository – Load compact –C –DAOS ON <PathNSFs>
  • 37. www.eknori.de Bug Alert !!! load compact -c –DAOS ON mail (w/o trailing backslash) enables DAOS on ALL applications, not only on those in mail subdirectory !! Documented Lotus Software Knowledge Base #1410926 SPR AHOE7XHM9J (fixed in 8.5.2)
  • 38. www.eknori.de The Result Design Text Tickets Design Text Attachments DAOS Repository
  • 39. www.eknori.de Transaction Protocol Trap load compact –c –DAOS ON NO attachment in DAOS repository tell daosmgr dbsummary count = 0 load compact -C -T -DAOS on
  • 40. www.eknori.de DAOS and replication DAOS is a server feature! On a client replica, all attachments are stored in the application. Replicas on a NON-DAOS server store attachments in the application Each server in a cluster has it„s own DAOS repository.
  • 41. www.eknori.de DAOS and replication (cont.)  Client  Server (as of 8.5.1) – „known” attachments are no longer transferred to server = „smart replication“ • Forward mail • Reply with history Server  Server (as of 8.5.1) – “known” attachments are no longer replicated to target server 8 new statistics – show stat DAOS
  • 42. www.eknori.de DAOS and replication (cont.) Domino 8.5 Domino 8.5.1 WARNING!!: ADMINP_ACCELERATED_REPLICA_OVERRIDE=4 !!
  • 43. www.eknori.de DAOS and replication (last slide )
  • 44. www.eknori.de DAOS and mail.box Remove RM_NO_LOG_OBJECTS_IN_MAILBOX from notes.ini. Enable DAOS on all mail.box files Enable DAOS on all mail files Increases performance
  • 45. www.eknori.de DAOS and Encryption Increases number of NLO files! – User A writes an encrypted mail with attachment to User B – User B writes an encrypted mail with same attachment to User A – Results in 4 NLO Files in repository Does not save space but separates static from dynamic data
  • 46. www.eknori.de DAOS and Quotas Technote: 1405456 Depends on the quota enforcement setting that is in place Check space used in file when adding a note – Acts on the physical size of the application (without attachments) – Possibly a problem when switching DAOS off!! • 64GB limit Check file size when extending the file Check file size when adding a note – Both acts on the logical size of the application (with attachments)
  • 47. www.eknori.de Keep daoscat.nsf in sync DON„T use DDM !! Use program document Tell daosmgr resync SPR# DROO7ZNPZ8 Added a new DAOSMGR resync switch to perform a quick resync. This new switch updates the DIT and DOI but does not run through the applications or updates the NLO refcounts. ( as of 8.5.1 IF1 )
  • 48. www.eknori.de „Needs Resync“ DDM points to the first application that causes the DAOS catalog to go into „Needs Resync“ state. If you cannot find a cause for the issue set debug_threadid=1 console_log_enabled=1 daos_logging=<path/daos.txt>STATE_CHANGE Many improvements in 8.5.2 that reduces the number of state changes (SPR# RCOR83NLQK)
  • 49. www.eknori.de DAOS resync time window feature The following two notes.ini parameters control when resync will be allowed to run – DAOS_RESYNC_START_TIME=12:00:00 AM – DAOS_RESYNC_STOP_TIME=04:00:00 AM Note that these parameters do not launch resync automatically; they just control whether or not resync may run at a given time, and also tell resync when to stop. A program document could be created to launch resync just after midnight. If the catalog is SYNCHRONIZED, then the resync will not take place. However, if a resync is needed, then the resync process will begin. When the end time is reached, the resync threads will finish up the databases they are processing and exit.
  • 50. www.eknori.de DAOS OFF What happens when DAOS is disabled / switched OFF? In application properties – No documents with attachments in the application – Application has documents with attachments • Status in Admin Client changes to „Read Only“ • DAOS still works for already stored attachments but no new attachments are written to the repository for the specific application. In server document – Same as 1
  • 51. www.eknori.de DAOS and missing NLO NLO has been deleted on the OS level TELL DAOSMGR LISTNLO MISSING <application.nsf> Output to <datadir> by default
  • 52. www.eknori.de Daosmgr in Domino 8.5.2 daosmgr options have been enhanced in Domino 8.5.2 You can now do a search for missing NLO on all applications in a subdirectory – tell daosmgr listnlo missing mail
  • 53. www.eknori.de DAOS and missing application NSF has been deleted on the OS level  Message on server startup • Informational - The DAOS catalog is not synchronized. Deletions will be postponed. Please run 'tell daosmgr resync' at the next convenient opportunity to re-synchronize. Has the application been deleted by accident? – NO • TELL DAOSMGR RESYNC oder RESYNC FORCE – YES • Restore from backup • Create a new replica from a cluster mate  Same behaviour when moving an application (on OS level)
  • 54. www.eknori.de Can we save another 50% If we disable encryption, then we could put NLO for all servers into a single repository … Yes, in theory It„s only experimental Does not work because each server has its own daoscat.nsf. Would cause problems with pruning
  • 55. www.eknori.de DAOS and Nagios ( Statistics ) Catalog.Status Repository.Size  Plugin
  • 56. www.eknori.de DAOS amd Statistics 8 new statistics (8.5.1) – show stat DAOS 8.5.1 Debug_NSF_Show_Allstats=1
  • 57. www.eknori.de DAOS and Statistics Windows: @echo off setlocal for /f "tokens=*" %%a in ('dir/w/s %1 ^|findstr "Datei(en)"') do ( set info=%%a ) echo %info% Linux: #!/bin/sh REPOSITORY=/local/daos OUTFILE=/local/daos.txt fCount=`find $REPOSITORY -type f | wc –l` fSize=`du -ksb $REPOSITORY` IFS=”/” array=($fSize) fDate=$(date +”%d-%m-%Y”) echo $fDate / $fCount / ${array[0]}>> $OUTFILE
  • 58. Use XML
  • 59. www.eknori.de DAOS and daos.cfg Contains information on file count and file size of .NLO files Created on server startup when it has been deleted Updated on server shutdown Do not backup or restore!!
  • 60. www.eknori.de DAOS and daoscat.nsf Contains index and refcount of NLO Contains index of all applications using DAOS Does NOT contain any view Re-Created on re-sync Created on server startup when it has been deleted Do not backup or restore!!
  • 61. www.eknori.de Admin Client ( Files tab ) New columns regarding DAOS – Logical size – Physical size – DAOS status – DAOS count – DAOS size New columns regarding compression
  • 62. www.eknori.de Backup & Restore No change in backing up NSF files. Additional backup job for NLO files NLO files do not require any special agent for backup IMPORTANT: the backup sequence – server is up during backup • NSF before NLO – server is down during backup • it‟s your choice IBM Technote # 1358548 - DAOS Backup and Restore
  • 63. www.eknori.de Resources DAOS Wiki • http://www- 10.lotus.com/ldd/dominowiki.nsf/archive?openview&title=DAOS&type=cat&c at=null&tag=DAOS DAOS Backup and Restore • http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daosbackup-and-restore DAOS Estimator • http://www.ibm.com/support/docview.wss?rs=463&uid=swg24021920 DAOS Technical Overview • http://www.nelotus.org/A55CBA/nelotus.nsf/87fde291d7608ada852564c9006 eeba5/b6523e7f4b15585b862575b5004d3488/$FILE/DAOSTechnicalOvervi ew_NELotus.pdf
  • 64. www.eknori.de This is it! Contact me at: ulrich.krause@eknori.de Thank you dank u wel merci Danke

×