Nsd, il tuo compagno di viaggio quando Domino va in crash

Autore: Fabio Pignatti Professione: Project Manager, Consulente Domino NSD, il tuo compagno di viaggio quando Domino va in crash

http://www.flickr.com/photos/jek-a-go-go/512926428/

Agenda  Crash e hang, perchè e percome  NSD, cos'è e perchè serve  Come decifrare un NSD  LND come strumento di diagnostica  Casi reali 3

http://www.flickr.com/photos/donnagrayson/195244498/

Crash e hang, le differenze  Crash  Hang • Domino non risponde ai client • Domino non risponde ai client • Domino non risponde da console • Domino risponde da console • Il sistema operativo è vivo • Il sistema operativo è vivo  NSD  NSD  Viene generato un NSD in  Non viene generato un NSD in automatico automatico ...  ... ma possiamo generarlo noi!

Le cause  Il codice delle applicazioni  Teniamo a bada le teste calde!  Database corrotti  Problemi HW (spazio disco, network)  Software terze parti  Controllare la compatibilità delle versioni! http://www.flickr.com/photos/cayusa/2569878338/ 6

http://www.flickr.com/photos/25508895@N00/181902056/

NSD, cos'è e perchè serve (1/2)  Cos'è?  NSD = Notes System Diagnostics  Dove si trova?  $DOMINO_DATAIBM_TECHNICAL_SUPPORT  Come si legge?  È un file ASCII  È suddiviso in diverse sezioni  Oggi ci focalizziamo su ...  NSD7 (da 6.5.5 in poi) ...  ... ma disponibile anche nelle precedenti 8

NSD, cos'è e perchè serve (2/2)  Perchè avere un NSD?  Strategia: dare strumenti di diagnostica per garantire continuità e affidabilità del servizio  Altri esempi: DDM, ADC  Ok ma nel pratico perchè serve NSD?  Consente di mettere a fuoco le cause del crash e potenzialmente di risolverle  Tornando alla stategia ...  #1233676 “NSD Fix List and NSD Update Strategy” http://www.flickr.com/photos/spacepotato/2450300968/ 9

Come decifrare un NSD – Le basi (1/4)  Traccia tutte le info di sistema  notes.ini  Processi  Memoria  File presenti nella domino directory 11

Come decifrare un NSD – Le basi (2/4)  Windows (nsd.exe)  nsd  nsd -detach (solo W2003 e XP)  GNU/Linux (nsd.sh)  nsd  Alcune opzioni  nsd -kill  nsd -batch (solo GNU/Linux)  nsd -info  nsd -monitor (solo W32) 12

Come decifrare un NSD – Le basi (3/4)  Formato del file  nsd_<Platform>_<ServerName>_YYYY_MM_DD@HH_MM_SS.log  Ummm ... e questi?  nsd_all_<Platform>_<Host>_MM_DD@HH_MM.log  nsd_kill*.log  nsd_info*.log (GNU/Linux)  sysinfo*.log (W32)  console*.log 13

Come decifrare un NSD – Le basi (4/4) 14

Come decifrare un NSD – Process Information (1/3)  Vengono riportati tutti i processi del sistema <@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@> Section: Notes Process Info (Time 07:34:25) <@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@> <@@ ------ Notes Process Info -> Load Modules for Process :: [ nSERVER: 0864] (Time 07:34:28) ------ @@> Base End Size Module 0x40000000 0x0043c000 245760 C:LotusDominonSERVER.EXE 0x7c800000 0x7c8c0000 786432 C:WINDOWSsystem32ntdll.dll 0x77e40000 0x77f42000 1056768 C:WINDOWSsystem32kernel32.dll  Nel dettaglio, le chiamate dei processi Domino <@@ ------ Notes Process Info -> Call Stack for Process :: [ nSERVER: 0864] (Time 07:34:28) ------ @@> ############################################################ ### thread 1/144: [ nSERVER: 0864: 0868] ### FP=0006f5a4, PC=7c8285ec, SP=0006f53c ### stkbase=00070000, total stksize=45056, used stksize=2756 ############################################################ [ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (1388,0,6f5c4,6009668a) [ 2] 0x77e424ed kernel32.Sleep+15 (1388,0,6f5d8,10003ba0) 15

Come decifrare un NSD – Process Information (2/3)  Parlando di crash ...  ... FATAL!  W32, vengono eseguiti 3 passi con dettaglio crescente  LINUX, 1 passo ############################################################ ### thread 14/144: [ nSERVER: 0864: 08d8] FATAL THREAD (Panic) ### FP=05e7ae30, PC=7c8285ec, SP=05e7adc0 ### stkbase=05e80000, total stksize=262144, used stksize=21056 ### EAX=0x00000201, EBX=0x77e42004, ECX=0x00000210, EDX=0x00017600 ### ESI=0x000003a4, EDI=0x00000000, CS=0x0000001b, SS=0x00000023 ### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00000297 ############################################################ ############################################################ [ 1] 0x7c8285ec ntdll.KiFastSystemCallRet+0 (3a4,493e0,0,5e7b3b0) [ 2] 0x77e61c8d kernel32.WaitForSingleObject+18 (3a4,493e0,3,5e7b5cc) (Panic) with STACK FRAMES [ nSERVER: ### PASS 2 : FATAL THREAD 0864: 08d8] ############################################################ @[ 3] 0x601a5754 nnotes.OSRunExternalScript@8+1284 (12c,1) # ---------- Dump of stack from SP to FP ---------- @[ 4] 0x601a5bea nnotes.FRTerminateWindowsResources+986 (1,0,1010,1) 05e7adc0 0B7D827C 1E1DE677 A4030000 00000000 | .}.| ...w .... .... | @[ 5] 0x601a5faf nnotes.OSFaultCleanupExt@24+895 (f44dd8,1010,0,0,0,5e7b8fc) @[ 6] 0x601a603a nnotes.OSFaultCleanup@12+26 (0,1010,0) 05e7add0 04AEE705 CCB5E705 10000000 0420E477 | .... .... .... . .w | 05e7ade0 24000000 01000000 00000000 00000000 | $... .... .... .... | @[ 7] 0x601b12a4 nnotes.OSNTUnhandledExceptionFilter@4+276 (5e7c934) @[ 8] 0x6017aca8 nnotes.Panic@4+520 (60bb0c80) 05e7adf0 00000000 00000000 00000000 00000000 | .... .... .... .... | 05e7ae00 00000000 00A22F4D FFFFFFFF 00E05DBD | .... ../M .... ..]. | @[ 9] 0x60002ae3 nnotes.LockHandle@12+323 (2d50,5e7c964,5e7c970) 05e7ae10 04AEE705 00000000 D4ADE705 A0B3E705 | .... .... .... .... | @[10] 0x6000f3b5 nnotes.OSMemGetSize@8+21 (60eed220,5e7ca34) @[11] 0x6002b468 nnotes.IDHTableValidate@8+24 (2d50,0) 05e7ae20 DCFFE705 601AE677 481DE677 00000000 | .... `..w H..w .... | ... @[12] 0x60025d6a nnotes.OSMemIDTableDestructor@4+26 (2d50) @[13] 0x608ec1e1 nnotes.DbFixup@4+9969 (5e7fb60) ... 16

Come decifrare un NSD – Process Information (3/3)  Ma sarà utile?  Aspettiamo il caso reale ############################################################ ### PASS 2 : FATAL THREAD with STACK FRAMES 19/20 [ namgr: 4756: 3424] ### FP=0d9ff63c, PC=60651e23, SP=0d9ff630, stksize=12 Exception code: c0000005 (ACCESS_VIOLATION) ############################################################ # ---------- Top of the Stack ---------- # 0d9ff630 00af0000 035a75c0 00000209 0d9ff678 |.....uZ.....x...| ... @[ 9] 0x62691192 nlsxbe.ANRTItem::ANRTEmbedObject+546 (5ae,0,f3fe74,0,d9ff25c,c003f4) # 0d9ffa90 0d9ffbe4 62694fc9 000005ae 00000000 |.....Oib........| # 0d9ffaa0 00f3fe74 00000000 00f3faec 0d326089 |t............`2.| # 0d9ffab0 0d9ffc64 0d326089 62775b7c 00000003 |d....`2.|[wb....| # 0d9ffac0 00000000 00f3fe40 00f3faec 0d296aa8 |....@........j).| # 0d9ffad0 62694ee0 00000014 07cd1e83 0d296aa8 |.Nib.........j).| # 0d9ffae0 00000000 0d296aa8 0d9ffb2c 0d36608c |.....j).,....`6.| # 0d9ffaf0 00000001 62694ee0 07cd1ffd 00000000 |.....Nib........| # 0d9ffb00 00000000 0d9ffb2c 0d36608c 0d296aa8 |....,....`6..j).| # 0d9ffb10 00000000 0d9ffcb4 00000000 00000088 |................| # 0d9ffb20 00000001 00000006 0d2e1118 6176614a |............Java| # 0d9ffb30 746f6c5f 645f7375 6e696d6f 6f6c5f6f |_lotus_domino_lo| # 0d9ffb40 5f6c6163 68636952 74786554 6d657449 |cal_RichTextItem| # 0d9ffb50 6d654e5f 4f646562 63656a62 32400074 |_NembedObject.@2| ... 17

Come decifrare un NSD – memcheck (1/5)  Ricca di informazioni utili: memoria condivisa, db, doc, view <@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@> Section: Notes Memory Analyzer (memcheck) (Time 07:34:58) <@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@> <@@ ------ Notes Memory -> Usage Summary -> Shared Memory Stats :: (Shared) (Time 07:34:59) ------ @@> TYPE : Count SIZE ALLOC FREE FRAG OVERHEAD %used %free Static-DPOOL: 131 528482304 510821364 17589916 0 99082 96% 3% VPOOL : 83 5069008 435807 4488228 0 146381 8% 88% POOL : 66 3558644 1853232 1657856 0 49676 52% 46% Overall : 131 528482304 504675280 23736000 0 295139 95% 4%  MM/OS Structure Information <@@ ------ Notes Data -> OS Data -> MM/OS Structure Information (Time 07:34:59) ------ @@> Start Time = 30/06/2008 07.31.40 Crash Time = 30/06/2008 07.34.18 Console Log Enabled = 1 Console Position = 561 Error Message = PANIC: LookupHandle: handle not allocated Console Position = 561 SharedDPoolSize = 4194304 FaultRecovery = 0x00010010 Cleanup Script Timeout= 300 Crash Limits = 3 crashes in 5 minutes StaticHang = Virtual Thread [ nSERVER: 0864: 0010] (Native thread [ nSERVER: 0864: 08d8]) (0x864/0x10/0x8d8) ConfigFileSem = ( SEM:#0:0x010d) n=0, wcnt=-1, Users=-1, Owner=[ : 0000] FDSem = ( RWSEM:#52:0x410f) rdcnt=-1, refcnt=0 Writer=[ : 0000], n=52, wcnt=-1, Users=0, Owner=[ : 0000] 18

Come decifrare un NSD – memcheck (2/5)  Open databases <@@ ------ Notes Data -> NSF Data -> Open Databases (Time 07:34:59) ------ @@> E:Notesdatadirdb.nsf Version = 43.0 SizeLimit = 0, WarningThreshold = 0 ReplicaID = 0xc1256fe0:0x004bb680 bContQueue = NSFPool [ 002aa365] Offline = No DeleteInProgress = No FDGHandle = 0xf0241910, RefCnt = 1, Dirty = Y DB Sem = (FRWSEM:0x0244) state=-1, waiters=0, refcnt=1, nlrdrs=0 Writer=[ nSERVER: 0864: 08b0] SemContQueue ( RWSEM:#0:0x029d) rdcnt=-1, refcnt=0 Writer=[ : 0000], n=0, wcnt=-1, Users=-1, Owner=[ : 0000] By: [ nSERVER: 0864: 0006] DBH= 339, User=CN=myserver/O=ACME 19

Come decifrare un NSD – memcheck (3/5)  Open Documents <@@ ------ Notes Data -> NSF Data -> Open Documents (BLK_OPENED_NOTE) (numDocs=21, numPools=21) :: [ nevent: 089c] (Time 07:35:00) ------ @@> DBH NOTEID HANDLE CLASS FLAGS IsProf IsAgent #Pools #Items Size Database 60 2310 0x0024 0x0001 0x0900 No No 1 60 14620 E:Notesdataddm.nsf ... 20

Come decifrare un NSD – memcheck (4/5)  Open views (globalmente) <@@ ------ Notes Data -> NIF Data -> NIF Collection Users (hash) (Time 07:34:59) ------ @@> CollUserVB hColl CollID CollectionVB Remote OFlags ViewNoteID Data HDB/Full View HDB/Full RefCnt SessionID Open By ------------ ------ ------ ------------ ------ ------ ---------- ------------- ------------- ------ ------------- -------------- [ 0004d005] 115 0 [ 00028005] NO 0x2822 630 140/140 140/140 1 [] [ : 0000] CollectionGroup: CurrentCollation = 0 [ 0006c805] 229 0 [ 00033005] NO 0x2822 598 201/201 201/201 1 [] [ : 0000] CollectionGroup: CurrentCollation = 0 [ 00068005] 194 0 [ 00033005] NO 0x2822 598 131/131 131/131 1 [] [ : 0000] CollectionGroup: CurrentCollation = 0 [ 000a6c05] 517 0 [ 00001005] NO 0x2822 270 320/320 320/320 1 [] [ nAMgr: 0c0c: 0004] 21

Come decifrare un NSD – memcheck (5/5)  Open views (specifico) <@@ ------ Notes Data -> NIF Data -> NIF Collections (Time 07:34:59) ------ @@> CollectionVB ViewNoteID UNID OBJID RefCnt Flags Options Corrupt Deleted Temp NS Entries ViewTitle ------------ ---------- -------- ------ ------ ------ -------- ------- ------- ---- --- ------- ------------ [ 00054005] 390 5f8514a1 18930 2 0x4200 00000008 NO NO NO YES 0 ($Profiles) CIDB = [ 000a7405] CollSem (FRWSEM:0x030b) state=0, waiters=0, refcnt=0, nlrdrs=0 Writer=[ : 0000] NumCollations = 1 bCollationBlocks = [ 00053805] bCollation[0] = [ 00055605] CollIndex = [ 00000014] Collation 0:BufferSize 15,Items 1,Flags 0 0: Ascending, by KEY, quot;$0quot;, summary# 1 ResponseIndex [ 02e3fcf0] NoteIDIndex [ 02e3fcf0] UNIDIndex [ 02e3fcf0] 22

Come decifrare un NSD – resource usage summary  La sezione inzia con <@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@> Section: Notes Memory Analyzer (memcheck) -> Resource Usage Summary (Time 07:35:01) <@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@>  VThread ** VThread [ nAMgr: 0bec: 0002] .Mapped To: PThread [ nAMgr: 0bec: 0bf0] .. SOBJ: addr=0x0124926c, h=0xf010403d t=0xc30a (BLK_LOOKUP_THREAD) .. SOBJ: addr=0x01235fdc, h=0xf0104016 t=0xc176 (BLK_SDKT) .. SOBJ: addr=0x01247774, h=0xf010403b t=0xc275 (BLK_NSFT) .. SOBJ: addr=0x01202d0c, h=0xf0104001 t=0xc130 (BLK_TLA) .. SOBJ: addr=0x012362f4, h=0xf0104018 t=0xc436 (BLK_LSITLS) .. Database: E:Notesdatadatabase.nsf .... DBH: 593, By: CN=myserver/O=ACME, WasAccessed=Yes .. Database: E:Notesdatadatabase2.nsf .... DBH: 498, By: CN=myserver/O=ACME, WasAccessed=Yes ...... doc: HDB= 498, NoteID= 770, hNote=0x0010, flags=0000, class=0200 [Agent Name=quot;EliminaPaquot;] .... DBH: 563, By: CN=Mario Rossi/O=ACME, WasAccessed=Yes .... DBH: 475, By: CN=Mario Rossi/O=ACME, WasAccessed=Yes ...... doc: HDB= 475, NoteID=2294, hNote=0x2d60, flags=0200, class=0001 ...... doc: HDB= 475, NoteID= 770, hNote=0x0016, flags=0000, class=0200 [Agent Name=quot;EliminaPaquot;] .. file: fd: 920, E:Notesdatadatabas2.nsf .. file: fd: 1112, E:Notesdatadatabase.nsf .. file: fd: 396, E:NotesdataIBM_TECHNICAL_SUPPORTconsole.log 23

http://www.flickr.com/photos/71502646@N00/2412677222/

LND come strumento di diagnostica (1/3)  Lotus Notes Diagnostic  È un database Notes integrato con il sistema operativo (W32)  2.6 alla data odierna  http://www-01.ibm.com/support/docview.wss?rs=899&uid=swg24019151 25

LND come strumento di diagnostica (2/3)  Come si usa 26

LND come strumento di diagnostica (3/3)  Cosa si ottiene?  Filtra l'NSD con le info più significative  Permette di navigare in modo “visuale” un NSD  Si collega alla KB IBM ed esegue per noi una ricerca!  Consente di copiare il thread con il fatal in modo veloce  Permette di confrontare vari NSD 27

http://www.flickr.com/photos/doctorow/2496308570/

Un caso reale: è lento (1/2)  Problema  Drrriiiiiiiiiiin!  “È lento! Aiuto!”  Ma sono passato la settimana scorsa e mi pareva performante ...  No, nessun cambiamento ... (ndr è in piedi da 2 gg la nuova procedura)  Si ma “quanto” è urgente?  MSPI 29

Un caso reale: è lento (2/2)  Generiamo un NSD durante il rallentamento  Nsd -detach (2003)  Analisi  LND, nessun crash  Ma aprendo il file ... ** VThread [ tomcat5: 1636: 5] .Mapped To: PThread [ tomcat5: 1636: 4484] .. SOBJ: addr=0x0ff83620, h=0xf010402a t=0xd1ba (PKG_LSBE+10) .. SOBJ: addr=0x0ff834a0, h=0xf010402b t=0xc130 (BLK_TLA) .. SOBJ: addr=0x00748b6c, h=0xf010402c t=0xc30a (BLK_LOOKUP_THREAD) .. SOBJ: addr=0x0ff83640, h=0xf010402d t=0xc275 (BLK_NSFT) .. Database: D:LotusDominoDatadb.nsf .... DBH: 304, By: Anonymous ...... doc: HDB=304, ID=2794, H=28, class=0001, flags=0101 .... DBH: 354, By: Anonymous ...... view: hCol=352, cg=N, noteID=302, someView|someView .... DBH: 344, By: Anonymous .... DBH: 269, By: Anonymous ...... doc: HDB=269, ID=2794, H=31, class=0001, flags=0101  Soluzione 30

Tranquilli c'è Jamie!  Risoluzione dei problemi di prestazioni di applicazioni Lotus Domino  Jamie Magee  In sala Michelangelo dalle 15:20 alle 16:00 31

http://www.flickr.com/photos/aspengull/111255602/

Un caso reale: è giù (1/4)  Problema  Drrriiiiiiiiiiin!  “È giù!”  ...  MSPI 33

Un caso reale: è giù (2/4)  Domino opportunamente configurato  Nsd generato in automatico  Analisi, LND  Crash rilevato. Stringhe di ricerca per la KB ma ... 34

Un caso reale: è giù (3/4)  ... nessun risultato trovato  Raffinando la ricerca:  SPR# BRIS5TXQJU - Fixed an HTTP crash during image conversion under certain circumstances. This problem rarely occurs in 6.5x but code changes make it more likely in 7.0.2 FP1.  SPR# COLY6Y2J3Q - Fixed an HTTP page rendering crash. This regression was introduced in 7.0.2 FP1. 35

Un caso reale: è giù (4/4)  Analisi, NSD ############################################################ ### FATAL THREAD 98/106 [ nhttp: 0270: 11c0] ### FP=0x4a4ba914, PC=0x60004141, SP=0x4a4ba910 ### stkbase=4a4c0000, total stksize=262144, used stksize=22256 ### EAX=0x4a4ba9d8, EBX=0x4a8e0b58, ECX=0x4a8e0b58, EDX=0x131182dc ### ESI=0x00000002, EDI=0x4a4baa84, CS=0x0000001b, SS=0x00000023 ### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010202 Exception code: c0000005 (ACCESS_VIOLATION) ############################################################ ... ############################################################ ### PASS 2 : FATAL THREAD with STACK FRAMES 98/106 [ nhttp: 0270: 11c0] ### FP=4a4ba914, PC=60004141, SP=4a4ba910 ### stkbase=4a4c0000, total stksize=262144, used stksize=22256 Exception code: c0000005 (ACCESS_VIOLATION) ############################################################ ... 4a4bab60 8C09A426 01000000 8C09A426 4CDF4B4A | ...& .... ...& L.KJ | 4a4bab70 34820310 FFFFFFFF 88AD4B4A FB070210 | 4... .... ..KJ .... | 4a4bab80 8010A426 8C09A426 436F6E74 656E742D | ...& ...& Cont ent- | 4a4bab90 54797065 3A20696D 6167652F 67696600 | Type : im age/ gif. | 4a4baba0 20636861 72736574 3D49534F 2D383835 | cha rset =ISO -885 | 4a4babb0 392D3100 70020000 01000000 7CAB4B4A | 9-1. p... .... |.KJ | http://www.flickr.com/photos/blude/2665906010/ 4a4babc0 14B07549 3A000000 8010A426 8010A426 | ..uI :... ...& ...& |  Soluzione 36

http://www.flickr.com/photos/razielss/1632964902/

Bonus track!  Test degli applicativi  ... non solo Domino!  Software terze parti  Manutenzione periodica degli archivi  Transaction logging  Censimento dei sistemi  ... non solo Domino!  Alcune accortezze che non guastano  Unread marks  Indici FT  Quota  KIS, keep it simple 38

Bonus track! 39

Bonus track! 40

Bonus track! 41

I Nostri Sponsor

Nsd, il tuo compagno di viaggio quando Domino va in crash

by Fabio Pignatti on Oct 16, 2008

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 43

Statistics

Nsd, il tuo compagno di viaggio quando Domino va in crash — Presentation Transcript

http://fabiopignatti.blogspot.com	36
http://www.slideshare.net	7