Compact, Compress, De-Duplicate (DAOS)

Compact, Compress, De-Duplicate Ulrich Krause DanNotes, 23. – 24. November 2011, Comwell Klarskovgaard, Korsør Denmark

About: Ulrich Krause
Administrator & Developer since 1993
Business Application Developer
[email_address] , Ratingen, Germany
OpenNTF http://openntf.org
!!HELP!!, Cebulon, Trigger Happy
Various XPages Custom Controls
IBM Champion 2011
https://www-304.ibm.com/connections/blogs/socialbusiness/entry/announcing_the_2011_ibm_champions_for_collaboration_solutions4?lang=de
Blog http://www.eknori.de
Notes Forum (http://www.atnotes.de)

How did I get here?

How did I get here?
Overall Distance 839 km
Flight SK 626
Düsseldorf  Kopenhagen
Train
Kopenhagen  Korsor
Car
Korsor  Venue
Thanks to Tobias !!

How is today‘s situation?
More and bigger documents and attachments
More expensive storage is needed
The average size of an application increases
Duplicate attachments ( n times )
Duration of standard tasks increases
More time for Backup
Network traffic increases
I/O and performance issues

What do we want to achieve?
Leverage existing storage
Reduce size of applications
Avoid attachment redundancies
Speed-up standard tasks
Reduce backup time
Reduce network traffic
Reduce disc I / O

Application model ( simplified )
Design
Documents
Text
Attachments
Embedded objects

Compact Secrets
DEBUG_ENABLE_COMPACT_8_5=1
W nn - Only compact databases which haven't been compacted successfully in the last nn days.
w - Exclude system databases (e.g., log.nsf, names.nsf).
X nn - Limit each database compaction to no more than nn minutes. (applies to in-place compaction only)
x nn - Limit total compaction time to nn minutes.
# nn - Execute up to nn compactions in parallel using different threads.
Don’t use more than 8 threads in parallel
ODS performs a copy-style compact only on databases with an ODS smaller than the ODS provided with this parameter.
ODS 48 would convert to ODS 51

Compact Secrets (Sample)
Load compact -# 8 mail
Compacts all applications in the mail subdir in 8 parallel Threads

Compact via application properties
Compacting a database does not necessarily reduce its size
If the application requires structural changes (typically a change of ODS [on-disk structure]). then a copy-style compaction will be performed (equivalent to -L).
If the application is logged (Transactional Logging enabled), then a simple in-place compaction will be performed (equivalent to -b).
If the application is not logged (Transactional Logging disabled), an in-place with file size reduction compaction will be performed (equivalent to -B).

Bug Alert!!!
In-place compact does not upgrade ODS when Create_R8_Databases is set in notes.ini
In Lotus Domino 8, a copy-style compact will automatically occur when a database needs a structural change, regardless of the compact options specified. However, you notice that databases compacted with the -b switch are not upgraded to ODS 48, even when Create_R8_Databases=1 is set in the notes.ini file.
Two notes.ini parameters are required to upgrade a database to ODS 48 using an in-place compact (compact -b):
1. Create_R8_Databases=1
2. Dont_Inplace_Compact_ODS43=1

Copy Style Compaction
The compact -C (copy-style compaction) (1) creates a new empty application container on the same server with the same replica ID and a .tmp suffix, (2) copies all of the documents from the original application to the new application container.
After all documents have been populated in the new application, (3) the old .nsf application is deleted and the (4) .tmp file is renamed to the original filename.

Defrag
File fragmentation saps Domino server I/O performance
Even a slightly fragmented file can cause a 50% decrease in non-cached read performance
Load compact –C does not allocate space needed before compacting the application
Applications are fragmented during copy-style compaction
Full Text Indexes are fragmented too
Even NLO files in DAOS repository are fragmented over the time
Use a tool to defrag your files speed up your Domino server !

Defrag
DefragNSF
Adam Osborne
http://www.preemptive.com.au/defrag
Domino Defrag
Andrew Luder, OpenNTF
http://www.openntf.org/Projects/pmt.nsf/ProjectLookup/DominoDefrag
Both products use the Microsoft defragmentation API

Defrag
Sample: Lotus Software KnowledgeBase ( 5 * compact –c )
Before
68 fragments
7550 ms
After
1 fragments
125 ms

Defrag
Domino Defrag uses IBM Lotus Notes C API
Open Source
FREE
There is also a professional support plan available
Well documented
32/64 Bit support
New in Version 3: Free Space Defragmentation
Uses jkdefragcmd.exe (http://kessels.com/jkdefrag/)

Domino Defrag
Xpages based administration application, logs, charts
Uses http://www.openntf.org/projects/pmt.nsf/ProjectLookup/Java%20Charts

FT Index and Defrag
When the index is updated it just about always fragments the index files
Little benefit for lots of work!
Adam Osbourne
http://ab1osborne.blogspot.com/2011/11/defragging-ft-indexes-little-benefit.html
http://ab1osborne.blogspot.com/2011/11/defragging-ft-indexes-part-2.html

Design Note Compression

New feature as of Domino 8
Needs ODS 48 or above
Create_R8_Databases=1
or Create_R85_Databases=1
load compact -c
load compact -c –n
Check “Compress database design”
Reduces size of design by 55 – 60 %
Reduces disc I / O

Document Body Compression

New feature as of Domino 8.0.1
Reduces size of Rich Text up to 60%
Needs ODS 48 or above
load compact -c –v
Check „Compress document data“

LZ1 Attachment Compression

As of release 6
Replaces Huffmann algorithm
NOT active when creating a new applications.
Recompress using
load compact -c –ZU
In 6.5.6 also set DEBUG_ENABLE_LZ1_REPAIR=1
To avoid problems with JAVA applets in <= 8.5 set
DEBUG_ENABLE_LZ1_HOST_TYPES=0
fixed in 8.5.1

There have been problems reported that the fulltext search does not work as expected using LZ1 compression.
To revert back to Huffmann use
Load compact –c –ZD
In 6.5.6 also set DEBUG_ENABLE_LZ1_REPAIR=1

Avoiding compression conversions
There are circumstances that compression conversion is explicitly avoid.
Overall, it is to avoid the performance penalty of attempting a compression and failing back to uncompressed for file types that are already compressed.
The common trick is applied to observe the file extensions and avoid these.
NONCOMPRESSABLE_FILES =<comma delimited replacement list>
Default list: zip,rar,jpg,gz,z,zz,lzo,zoo,rz,lzma,bz2

LZ1 Attachment Compression Design Text Attachments

DAOS Estimator

DAOS Estimator
Free tool to plan your DAOS rollout
Download: http://www-01.ibm.com/support/docview.wss?rs=463&uid=swg24021920

DAOS Estimator (notes.ini)
DAOS Estimator by default has a large gap in attachment size evaluation between 64k and 1 MB
DAOSEST_BUCKETS=16,64,128,256,512,768,1024,2048,3072,4096
The above line sets to:
16K, 64K, 128K, 256K, 512K, 768K, 1MB, 2MB, 3MB and 4MB
Source: Lotus Software Knowledge Base
http://www-01.ibm.com/support/docview.wss?rs=0&q1=1418102&uid=swg21418102

LZ1 != HUFFMAN
Db1 – LZ1 enabled
Db2 – LZ1 disabled
Result is 2 .NLO Files !!

D A O S

Detailled scientific study

DAOS
Domino Attachment Object Service
Attachments are no longer stored in application
Attachments are stored as NLO (Notes Large Object )
Document contains only a ticket
Separates static and dynamic data
Only one copy of a bit-like attachment
Transparent to users and 3rd-Party Products

Prerequisites
Domino 8.5 ( ODS 51)
SHARED MAIL OFF
Transaction protocol
Create DAOS Repository
Activate DAOS on server
Activate DAOS on application

Switch OFF Shared Mail
Serverdocument – Shared Mail
Field Shared Mail = None
„ Load object unlink shared.nsf“

Transaction Protocol
Server document – Transactional Logging
Required for DAOS – no discussion!!
Use a separate disk drive.
Create log file folder
Choose method
Circular
4GB limit
Archive
No limit, backup will delete logs
Linear
Like circular, NO size limit
CREATE_R85_LOG=1
Restart server

DAOS Repository
Use separate disk drive
Can be SAN, NAS
Each server has it‘s own repository
Max.1.000 subdirectories
Max 40.000 .NLO / subdirectory
= 40.000.000 unique .NLOs
Daos.cfg ( in Domino Data Directory)
Daoscat.nsf
.NLO files are encrypted with server.id
DAOS_ENCRYPT_NLO=0
Include repository in backup!!

DAOS Repository
How do you change the number of DAOS NLO's per directory?
DAOS_MAX_FILES_PER_SUBCONTAINER=40000
Requires a server restart
Will only affect the current DAOS subdirectory (older subdirectories are not changed).
This will also affect the maximum number of files in the DAOS store which is currently this values multiplied by 1000 (this value is not configurable currently).
1.000 * DAOS_MAX_FILES_PER_SUBCONTAINER = MAX_NLO

Activate DAOS on server
Goto Serverdocument – DAOS
Set Minimum size of an object
recommended 64kB (= DEFAULT in 8.5.1 )
Evaluate with DAOS Estimator
Set DAOS base path
Put it outside the data path!
Set Delete objects in repository
Depends on your backup strategy
0 Days = immediate delete after the refcount = 0
Manually from console: TELL DAOSMGR PRUNE 0
Restart the server
show server

Activate DAOS on applications
Application  properties  „propeller hat“ for single application
Use Admin client for multiple applications
Valid for new attachments only!
To move existing attachments into the DAOS repository
Load compact –C –DAOS ON <PathNSFs>

Bug Alert !!!
load compact -c -DAOS ON mail (w/o trailing backslash) enables DAOS on ALL applications, not only on those in mail subdirectory !!
Documented Lotus Software Knowledge Base #1410926
SPR AHOE7XHM9J ( fixed in 8.5.2, no regression in 8.5.3 )

Bug Alert !!!
"tell daosmgr status path/DatabaseName.nsf„
Notice the forward slash (/) instead of a back slash ()
A programming error was found but will not be corrected. It will be a permanent restriction. Use the correct syntax
LO50060: ENHANCEMENT: "THIS DATABASE IS NOT REGISTERED WITH DAOS"
tell daosmgr status path/DatabaseName.nsf tell daosmgr status path/DatabaseName.nsf

The Result

Transaction Protocol Trap
load compact –c –DAOS ON
NO attachment in DAOS repository
tell daosmgr dbsummary count = 0
load compact -C -T -DAOS on

DAOS and replication
DAOS is a server feature!
On a client replica, all attachments are stored in the application.
Replicas on a NON-DAOS server store attachments in the application
Each server in a cluster has it‘s own DAOS repository.

DAOS and replication (cont.)
Client  Server (as of 8.5.1)
„ known” attachments are no longer transferred to server = „smart replication“
Forward mail
Reply with history
Server  Server (as of 8.5.1)
“ known” attachments are no longer replicated to target server
8 new statistics
show stat DAOS

DAOS and replication (cont.)
Domino 8.5
Domino 8.5.1
WARNING!! : ADMINP_ACCELERATED_REPLICA_OVERRIDE=4 !!

DAOS and replication (last slide  )

DAOS and mail.box
Remove RM_NO_LOG_OBJECTS_IN_MAILBOX from notes.ini.
Enable DAOS on all mail.box files
Enable DAOS on all mail files
Increases performance

DAOS and Encryption
Increases number of NLO files!
User A writes an encrypted mail with attachment to User B
User B writes an encrypted mail with same attachment to User A
Results in 4 NLO Files in repository
Does not save space but separates static from dynamic data

DAOS and Quotas
Technote: 1405456
Depends on the quota enforcement setting that is in place
Check space used in file when adding a note
Acts on the physical size of the application (without attachments)
Possibly a problem when switching DAOS off!!
64GB limit
Check file size when extending the file
Check file size when adding a note
Both acts on the logical size of the application (with attachments)

Keep daoscat.nsf in sync
DON‘T use DDM !!
Use program document
Tell daosmgr resync
SPR# DROO7ZNPZ8 Added a new DAOSMGR resync switch to perform a quick resync. This new switch updates the DIT and DOI but does not run through the applications or updates the NLO refcounts. ( as of 8.5.1 IF1 )

„ Needs Resync“
DDM points to the first application that causes the DAOS catalog to go into „Needs Resync“ state.
If you cannot find a cause for the issue set
debug_threadid=1 console_log_enabled=1 daos_logging=<path/daos.txt>STATE_CHANGE
Many improvements in 8.5.2 that reduces the number of state changes (SPR# RCOR83NLQK)

DAOS resync time window feature
The following two notes.ini parameters control when resync will be allowed to run
DAOS_RESYNC_START_TIME=12:00:00 AM
DAOS_RESYNC_STOP_TIME=04:00:00 AM
Note that these parameters do not launch resync automatically; they just control whether or not resync may run at a given time, and also tell resync when to stop.
A program document could be created to launch resync just after midnight. If the catalog is SYNCHRONIZED, then the resync will not take place. However, if a resync is needed, then the resync process will begin. When the end time is reached, the resync threads will finish up the databases they are processing and exit.

DAOS OFF
What happens when DAOS is disabled / switched OFF?
In application properties
No documents with attachments in the application
Application has documents with attachments
Status in Admin Client changes to „Read Only“
DAOS still works for already stored attachments but no new attachments are written to the repository for the specific application.
In server document
Same as 1

DAOS and missing NLO
NLO has been deleted on the OS level
TELL DAOSMGR LISTNLO MISSING <application.nsf>
Output to <datadir> by default

Daosmgr in Domino 8.5.2
daosmgr options have been enhanced in Domino 8.5.2
You can now do a search for missing NLO on all applications in a subdirectory
tell daosmgr listnlo missing mail

DAOS and missing application
NSF has been deleted on the OS level
Message on server startup
Informational - The DAOS catalog is not synchronized. Deletions will be postponed. Please run 'tell daosmgr resync' at the next convenient opportunity to re-synchronize .
Has the application been deleted by accident?
NO
TELL DAOSMGR RESYNC oder RESYNC FORCE
YES
Restore from backup
Create a new replica from a cluster mate
Same behaviour when moving an application (on OS level)

Can we save another 50%
If we disable encryption, then we could put NLO for all servers into a single repository …
Yes, in theory
It‘s only experimental
Does not work because each server has its own daoscat.nsf.
Would cause problems with pruning

DAOS and Nagios ( Statistics )
Catalog.Status
Repository.Size
Plugin
http://www.eknori.de/2009-07-11/nagios-centreon-and-daos/ http://www.eknori.de/2009-07-12/more-nagios-more-daos/

DAOS and Statistics
8 new statistics (8.5.1)
show stat DAOS
8.5.1 Debug_NSF_Show_Allstats=1

DAOS and Statistics
Windows:
@echo off
setlocal
for /f "tokens=*" %%a in ('dir/w/s %1 ^|findstr "Datei(en)"') do (
set info=%%a
)
echo %info%
Linux:
#!/bin/sh
REPOSITORY=/local/daos
OUTFILE=/local/daos.txt
fCount=`find $REPOSITORY -type f | wc –l`
fSize=`du -ksb $REPOSITORY`
IFS=”/”
array=($fSize)
fDate=$(date +”%d-%m-%Y”)
echo $fDate / $fCount / ${array[0]}>> $OUTFILE

Use XML

DAOS and daos.cfg
Contains information on file count and file size of .NLO files
Created on server startup when it has been deleted
Updated on server shutdown
Do not backup or restore!!

DAOS and daoscat.nsf
Contains index and refcount of NLO
Contains index of all applications using DAOS
Does NOT contain any view
Re-Created on re-sync
Created on server startup when it has been deleted
It should not be replicated, compacted, indexed, or otherwise manipulated

DAOS and daoscat.nsf
You receive an error when opening daoscat.nsf
Error, special noteid not found
Database is not fully initialized yet. (It does not contain any views.) you will have to use 'Replicate...' on the 'File, Replication' menu to initialize it.
The operation that incurred the error should not be performed
("Doctor, it hurts when I do this" "Then don't do that")

Admin Client ( Files tab )
The application now has sizes
Logical size
Physical size
DAOS status
DAOS count
DAOS size
Quotas use the logical size.
New columns regarding compression

Backup & Restore
No change in backing up NSF files.
Additional backup job for NLO files
NLO files do not require any special agent for backup
IMPORTANT : the backup sequence
server is up during backup
NSF before NLO
server is down during backup
it’s your choice
IBM Technote # 1358548 - DAOS Backup and Restore

FT Indexes in a different location
Migrate existing FT Indexes by bringing down server, moving all the indexes into matching subdirectory in new FTBasePath = and restart Domino Server
Works on the Notes Client as well.
Run updall -f to rebuild all FT indexes

FT Indexes in a different location
You can move FTs to local drives, while the dbs are located on a network share/homeshare, e.g. for Citrix XenApp installations
If you set the notes.ini and don't move the indexes, a new index will be built from scratch in the new location and the old index will be deleted... No migration but fairly user friendly when it comes to cleanup..
If for some reason the FTBasePath = set in the notes.ini does not exist or server does not have proper access to, the code defaults back to the same location as the database as it always did.

ODS Tuning
Automatic Client ODS Update
As of release 8.5.2
Can be configured in „Desktop Policy
Desktop Policy  Mail  MailSettings
Or via notes.ini NSF_UpdateODS=1 + (CREATE_R85_DATABASES=1)
NSF_UpdatedToODS=51
Applications that are in use will be upgraded on next client startup

ODS Tuning
Technote : Upgrading multiple local databases to a new ODS
Unfortunately the admin help that is delivered with 8.5.2 does not include any information about this feature.
https://www-304.ibm.com/support/docview.wss?uid=swg21429889

ODS Tuning
ns6, ns7 … instead of .nsf
compact -c does not change the ODS !!
Even, if you revert the suffix to .nsf
Technote: 1086780
Create new copy or replica
Notespeek Download : https://www-304.ibm.com/support/docview.wss?uid=swg24005686

ODS Tuning
Upgrading the ODS of templates
load compact -c
Upgrades all.nsf files, but not templates
load compact –c *.ntf
ERROR: "Database compactor error: File does not exist.„
load compact –c cois_crm.ntf
Works
Workaround: use .ind files
dir *.ntf /b > templates.ind
load compact -c templates.ind

Agent Manager Retry on Compact
There is a new setting starting with 8.5.3 and 8.5.2 FP3 to retry to run an agent after the agent has been blocked out thru a compact previously.
SPR# KMUR63DF3V - Fix introduces an ini DEBUG_AMGR_ENABLE_RETRY_ON_COMPACT to allow an agent to run on a time interval once database compact is complete. Previously when a database was being compacted and an attempt was made to load a scheduled agent it would fail and the agent would be marked to not run again unless the user restarted the agent manager or the agent cache refreshed. This fix introduces a notes.ini variable to allow the agent in question to be retried on it's subsequent time interval.

JavaScript/CSS Aggregation
Dynamically aggregates multiple Dojo modules, or multiple CSS into a single file.
This results in the following performance improvements:
a decrease in requests sent from the browser to the server
an increase in user performance, particularly in the context of networks with high latency
an increase in the speed of JS/CSS parsing from the browser.
the freeing up of server connections to fulfill other requests.
Server level in xsp.properties: xsp.resources.aggregate=true

JavaScript/CSS Aggregation

XPages Preload
New notes.ini variable as of release 8.5.3
Server and Client
XPagesPreload=1
LoadsXPages runtime components at server startup
XPagesPreloadDB =Server!!Db.nsf/XPage.xsp,myLocalDb.nsf
Loads XPages runtime Java classes for 1 to n applications

Resources
DAOS Wiki
http://www-10.lotus.com/ldd/dominowiki.nsf/archive?openview&title=DAOS&type=cat&cat=null&tag=DAOS
DAOS Backup and Restore
http://www-10.lotus.com/ldd/dominowiki.nsf/dx/daosbackup-and-restore
DAOS Estimator
http://www.ibm.com/support/docview.wss?rs=463&uid=swg24021920
DAOS Technical Overview
http://www.nelotus.org/A55CBA/nelotus.nsf/87fde291d7608ada852564c9006eeba5/b6523e7f4b15585b862575b5004d3488/$FILE/DAOSTechnicalOverview_NELotus.pdf

Har du nogen spørgsmål
Mail: [email_address]
Blog: http://www.eknori.de
Sametime: bleedyellow.com
Twitter: eknori
Tak for
jeres opmærksomhed

Compact, Compress, De-Duplicate (DAOS)

by Ulrich Krause on Nov 24, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Compact, Compress, De-Duplicate (DAOS) — Presentation Transcript