Compact, Compress, De-Duplicate (DAOS)
•Download as PPT, PDF•
4 likes•9,387 views
The document discusses techniques for compacting, compressing, and de-duplicating data in Domino applications to reduce storage usage and improve performance. It covers compacting databases, compressing design elements, documents, and attachments, using DAOS to store attachments externally, and tools for defragmenting files.
![About: Ulrich Krause ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Compact, Compress, De-Duplicate (DAOS)
Similar to Compact, Compress, De-Duplicate (DAOS) (20)
More from Ulrich Krause
More from Ulrich Krause (14)
Recently uploaded
Recently uploaded (20)
Compact, Compress, De-Duplicate (DAOS)
- 1. Compact, Compress, De-Duplicate Ulrich Krause DanNotes, 23. – 24. November 2011, Comwell Klarskovgaard, Korsør Denmark
- 3. How did I get here?
- 4. How did I get here?
- 30. LZ1 Attachment Compression Design Text Attachments
- 31. DAOS Estimator
- 35. D A O S
- 47. The Result
- 52. DAOS and replication (last slide )
- 67. Use XML
Editor's Notes
- In Deutschland wurde am 3. August 1984 um 10:14 Uhr MEZ die erste Internet-E-Mail empfangen: Michael Rotert von der Universität Karlsruhe (TH) empfing unter seiner Adresse „rotert@germany“ eine Grußbotschaft von Laura Breeden („breeden@scnet-sh.arpa“) an der US-amerikanischen Plattform CSNET aus Cambridge (Massachusetts) zur elektronischen Kommunikation von Wissenschaftlern, die einen Tag zuvor (am 2. August 1984, 12:21 Uhr)[8] abgeschickt worden war. Eine Kopie dieser E-Mail wurde erst später an den Leiter des Projekts, Werner Zorn („zorn@germany“), geschickt.[9]
- There are circumstances that compression conversion is explicitly avoid. Overall, it is to avoid the performance penalty of attempting a compression and failing back to uncompressed for file types that are already compressed. The common trick is applied to observe the file extensions and avoid these. Default list: zip,rar,jpg,gz,z,zz,lzo,zoo,rz,lzma,bz2 Ini to control: NONCOMPRESSABLE_FILES=<comma delimited replacement list> There is also a mechanism to avoid for specific item names. Default List: $JavaFile Init to control: NONCOMPRESSABLE_NOTE_ITEMS=<comma delimited replacement list - and do not remove $JavaFile - see below> For $JavaFile problem see technote: https://www-304.ibm.com/support/docview.wss?uid=swg21370344
- The reason for my concern? We have been told that the DAOS store has lesser disk performance demands than say the Transaction Logs or the data directory, yet it appears that when a DAOS MGR RESYNC or similar operation is initiated, actually a very intensive audit of the DAOS area takes place. This is where the 40,000 file per directory comes in... Just doing an 'ls' or 'dir' of a very large directory like that takes a significant time, particularly when this area is stored on NFS or iSCSI disk. In the case of a NFS mounted DAOS directory (say on an enterprise-class NetApps file), that 40000 files per subcontainer is too much. When the DAOS MGR RESYNC starts, it enumerates the contents of all the subcontainers. Due to the nature of NFS this is an extremely intensive operation. For 1.2 million attachments in 30 subcontainers this takes quite some time on NFS (3 hours or so) and is a heavy hit on the NFS filer performance. During this time, the daos catalogue daos cat.nsf is &quot;unavailable&quot;, and so mail routing cannot occur until directory enumeration is complete. This is only going to get worse as more attachments are added to the DAOS store. Whenever the DAOS is resync'd, mail cannot be routed until this initial phase is completed and it starts on the databases themselves. In my opinion, we would see much better performance on initial resync if file count per subcontainer can be restricted to 5000 files or less for situations such as NFS. So if there isn't a means of decreasing the files/directory ratio now, can one be created? Feedback number SMIE82TFFY created by Stuart McIntyre on 19.02.2010
- http://www-01.ibm.com/support/docview.wss?uid=swg21449104 Question How does the DAOS resync time window feature work. Answer Brief explanation of how the Resync Time Window feature works The following two notes.ini parameters control when resync will be allowed to run. Note that these parameters do not launch resync automatically; they just control whether or not resync may run at a given time, and also tell resync when to stop. DAOS_RESYNC_START_TIME=12:00:00 AM DAOS_RESYNC_STOP_TIME=04:00:00 AM The format of the time is as follows: HH:MM:SS AM/PM The above example will allow resync to run between midnight and 4:00 AM. A program document could be created to launch resync just after midnight. If the catalog is SYNCHRONIZED, then the resync will not take place. However, if a resync is needed, then the resync process will begin. When the end time is reached, the resync threads will finish up the databases they are processing and exit. If resync finished processing all databases during the time window, then the catalog will go to the SYNCHRONIZED state. At this point there is still a little more work to be done to the catalog. Even though the time window is closed, this processing will be finished up. Therefore, there is a potential for resync to run past the end time, but the remaining work is low-impact and will not effect server response time. Note that until resync scans all databases on the system and puts them into the SYNCHRONIZED state, the DAOS catalog will remain in the RESYNCING state. This indicates that resync still has work to be done. It does not, however, indicate that resync is currently running. Resync Time Window FAQ: 1. Why does processing continues after set stop time, DAOS_RESYNC_STOP_TIME? One reason for the resync to continue is that each thread will finish the NSF it is processing, and this work may run past the specified stop time. The next night, when resync is run again, it will continue with the next databases to be processed. Another reason might be that after the nlo file reference counts for all the databases are completed, the remaining processing will be low-cost in terms of time, and will be allowed to complete. 2. What happens if DAOS_RESYNC_START_TIME and DAOS_RESYNC_STOP_TIME are set using the 24 hour clock? 24 hour clock times are not currently supported for this feature. The issue can be tracked under the SPR SRIO84NQTY. The following example shows the behavior of the time window with the 24-hour clock set. If you see this behavior, set the DAOS_RESYNC_START_TIME/DAOS_RESYNC_STOP_TIME to supply values based on the 12-hour clock. A. set config DAOS_RESYNC_START_TIME=15:00:00 B. set config DAOS_RESYNC_STOP_TIME=16:00:00 C. tell daosmgr resync results in an abort. sh config DAOS_RESYNC_START_TIME [602228:00009-01286] DAOS_RESYNC_START_TIME=15:00:00 sh config DAOS_RESYNC_STOP_TIME [602228:00009-01286] DAOS_RESYNC_STOP_TIME=16:00:00 tell daosmgr resync [376974:00002-00001] Current Time is not within resync time slot, aborting... tell daosmgr status [376974:00002-00001] 04/19/2010 15:36:42 DAOSMGR: Status started [376974:00002-00001] 04/19/2010 15:36:42 threadsShouldQuit = FALSE [376974:00002-00001] 04/19/2010 15:36:42 refcount = 1 [376974:00002-00001] 04/19/2010 15:36:42 threadsCreated = 3 [376974:00002-00001] 04/19/2010 15:36:42 [376974:00002-00001] 04/19/2010 15:36:42 basePath = /local1/notesdata/ [376974:00002-00001] 04/19/2010 15:36:42 baseStoragePath = /local1/notesdata/DAOSREP1/ [376974:00002-00001] 04/19/2010 15:36:42 storagePath = /local1/notesdata/DAOSREP1/ [376974:00002-00001] 04/19/2010 15:36:42 pruneInterval = 0 [376974:00002-00001] 04/19/2010 15:36:42 maxFilesPerSubcontainer = 40000 [376974:00002-00001] 04/19/2010 15:36:42 maxSubcontainers = 1000 [376974:00002-00001] 04/19/2010 15:36:42 minObjectSize = 4096 [376974:00002-00001] 04/19/2010 15:36:42 catalogState = NEEDS RESYNC [376974:00002-00001] 04/19/2010 15:36:42 catalogVersion = 3 [376974:00002-00001] 04/19/2010 15:36:42 catalogMinVersion = 5 3. Why does DAOSMGR report that a resync has completed during shutdown? A series of messages similar to those following indicates that the resync thread has completed processing and has terminated. It is not an indication that the resync process has completed its task of synchronizing the DAOS catalog. > quit 05/24/2010 08:53:17 AM Cluster Database Directory Terminating 05/24/2010 08:53:17 AM Cluster Database Directory shutdown ... 05/24/2010 08:53:18 AM PID=3888 TID=5 RESYNC Object rebuild finished: 0x0000 05/24/2010 08:53:18 AM PID=3888 TID=5 Unknown Failed rebuilding the object entries/index in the DAOS catalog: 0000 05/24/2010 08:53:18 AM DAOS was unable to rebuild the list of external files while trying to resynchronize. 05/24/2010 08:53:18 AM There was an error rebuilding the catalog. 05/24/2010 08:53:18 AM PID=3888 TID=5 RESYNC Catalog rebuild finished: 0x0AC5 05/24/2010 08:53:18 AM PID=3888 TID=5 RESYNC DAOS Resync 'QUICK' mode, exiting without updating the ref counts 05/24/2010 08:53:18 AM The DAOS catalog cannot be resynchronized. DAOS deletions will be postponed. 05/24/2010 08:53:18 AM PID=3888 TID=5 RESYNC Resync catalog done. 05/24/2010 08:53:18 AM DAOSMGR: Resync completed 05/24/2010 08:53:20 AM DAOSMGR: DAOS Manager terminating 4. Why doesn't the console message for DAOS resync aborting include the window of time? If the time window is set and the current time is not within the window, a warning will report that the current time is not within the resync time window, but there will be no information on when the time window is. The same behavior will occur for the &quot;tell daosmgr resync quick&quot; command. The issue with the time window not being reported in this warning can be tracked under the SPR SRIO84NKUU. sh config DAOS_RESYNC_START_TIME [069C:0018-0C10] DAOS_RESYNC_START_TIME=08:00:00am sh config DAOS_RESYNC_STOP_TIME [069C:0018-0C10] DAOS_RESYNC_STOP_TIME=09:00:00am tell daosmgr resync [0E10:0002-0DE0] Current Time is not within resync time slot, aborting... [009C:0002-0098] 04/19/2010 11:14:13 AM Admin Process: Searching Administration Requests database 04/19/2010 11:14:14 AM Agent Manager: Error validating execution rights for agent 'AAA Test Validate' in database 'dct\\lsbeexpresults.nsf'. Agent signer 'Agent Signer/Agents', effective user 'Agent Signer/Agents'. Examine 'Programmability Restrictions' However, you can use &quot;tell daosmgr status&quot; to report the time. 05/24/2010 08:53:09 AM resyncWindowStartTime = 08:00:00 PM 05/24/2010 08:53:09 AM resyncWindowEndTime = 10:00:00 PM 5. How does an administrator start the resync outside the currently set window? You can accomplish this in two ways: 1. Reset the time window using &quot;set config DAOS_RESYNC_START_TIME=<current time>&quot; and &quot;set config DAOS_RESYNC_STOP_TIME=<time to stop>&quot; and restart the resync by issuing the &quot;tell daosmgr resync&quot; command. If necessary, reset the time window back to what it its original setting. 2. You can considering using the FORCE option, &quot;tell daosmgr resync quick force&quot; or &quot;tell daosmgr resync force&quot;. It will force a resync outside of resync time window. However, this method is less desirable because there is a time cost associated with the FORCE commands. The resync FORCE will result in aborting all the work done to synchronize the catalog up until that point and will start all over. 6. Why is Resync aborted immediately when a Resync Time Window is set? This could happen if the stop and start time were mistakenly swapped. Consider the following time window: DAOS_RESYNC_START_TIME=10:00AM DAOS_RESYNC_STOP_TIME=8:00AM If resync is started at all after 10AM, it stops immediately because the time check shows that the current time is after the stop time of 8AM.
- XPages Preload options New options designed to improve the startup performance of XPages applications have been added to this release. A preference can be set that preloads the XPages runtime on the Domino server and Notes client. Additionally, specific XPages applications may also be preloaded. The preferences are set as entries in the NOTES.INI file and have the following form: XPagesPreload=1 XPagesPreloadDB=myServer!!myDb.nsf/myXPage.xsp,myLocalDb.nsf/foo.xsp The first entry (XPagesPreload) causes an extensive list of XPages runtime Java classes to be preloaded, so they are available in memory when an application is opened by a user. The second preference (XPagesPreloadDB) allows a comma separated list of applications to be specified, with or without a specific XPage extension. Again, any required classes are loaded when the server or client starts up and are thus available as soon as any declared application is accessed by a user. The benefit of this may be more evident when running in remote mode on the Notes client, i.e. the XPages is running on the client but the application itself resides on a remote server. In this instance, the Java classes in the application (XPages, custom controls) need to be loaded across the network in order to be executed in the local XPD web container. Preloading such applications produces more responsive initial user experience.