Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

D3 Troubleshooting

198 views

Published on

Provide training on how to troubleshoot Rocket® D3 DBMS on both Windows and UNIX platforms. Focusing on utilities and commands utilized to troubleshoot D3 in areas of performance, license management, overflow management, process locking, D3 server failures and error reporting and management.

Published in: Software
  • Be the first to comment

D3 Troubleshooting

  1. 1. 1 Rocket® D3 DBMS Troubleshooting John Bramley, Senior Director, MV Lab
  2. 2. 2 Abstract  Provide training on how to troubleshoot Rocket® D3 DBMS on both Windows and UNIX platforms. Focusing on utilities and commands utilized to troubleshoot D3 in areas of performance, license management, overflow management, process locking, D3 server failures and error reporting and management. ©2015 Rocket Software, Inc. All Rights Reserved.
  3. 3. 3 Credits and Acknowledgements Developers • James Rettally, Senior Software Engineer • Timothy Rude, Senior Manager, Software Engineering Co-Presenter • Mark Locke, Manager, Quality Control ©2015 Rocket Software, Inc. All Rights Reserved.
  4. 4. 4 Agenda ©2015 Rocket Software, Inc. All Rights Reserved. Performance License Management Overflow Management Lockups (hangs) Crashes
  5. 5. 5 Troubleshooting Tools
  6. 6. 6 Best Tool Ever – What Changed? Application • Application updated or new module added? • Code changes to a specific subroutine or module? • Interface added – For example, new web portal utilizing different interface to D3? • Additional application load – D3 users added? ©2015 Rocket Software, Inc. All Rights Reserved.
  7. 7. 7 Best Tool Ever – What Changed? D3 • Version upgrade or patches installed? Operating System • Upgrade or update (for example, Windows updates applied or Linux packages upgraded)? • Additional software installed outside of the D3 environment? • Changes to the network or storage? ©2015 Rocket Software, Inc. All Rights Reserved.
  8. 8. 8 Best Tool Ever – What Changed? Virtual Environment • Additional virtual instance added to the environment? • Changes made to the resources for the virtual instance (memory, CPU, network, etc.)? Physical Hardware • Upgrade or update? ©2015 Rocket Software, Inc. All Rights Reserved.
  9. 9. 9 Troubleshooting Tools – D3 TCL check.pib d3watcher watch buffers (Unix only) ©2015 Rocket Software, Inc. All Rights Reserved.
  10. 10. 10 Troubleshooting Tools – UNIX Commands gdb strace /proc top or topas (AIX) Note – above commands should be run as the ‘root’ user ©2015 Rocket Software, Inc. All Rights Reserved.
  11. 11. 11 Troubleshooting Tools – Window Commands Performance Monitor Resource Monitor Handle ProcDump Process Explorer ©2015 Rocket Software, Inc. All Rights Reserved.
  12. 12. 12 Performance
  13. 13. 13©2015 Rocket Software, Inc. All Rights Reserved. Performance What changed? Identify the bottleneck Identify the source Identify the cause
  14. 14. 14 Identify the Bottleneck  CPU • Windows - nt_pinfo (l, Task Manager Unix - top  Disk • Windows - Task Manager Unix – iostat  Memory • Windows - nt_pinfo, Task Manager Unix – top, buffers  Lock Contention • list-locks, list-lockq • monitor debugger cs (Windows), gl (Unix) ©2015 Rocket Software, Inc. All Rights Reserved.
  15. 15. 15 Identify the Source Single or multiple lines? Phantom? External process? Network? ©2015 Rocket Software, Inc. All Rights Reserved. NAS? Poorly sized file? Other?
  16. 16. 16 Identify the Cause Drill down from what was found to be the source • Program doing loop; for example, system(14) without a sleep • Extra network traffic added, affecting disk performance • Data has grown, gradually causing larger memory requirements • More users accessing data remotely (new mobile app) adding to lock contention D3 problem • Provide reproducible case, if possible ©2015 Rocket Software, Inc. All Rights Reserved.
  17. 17. 17 License Management
  18. 18. 18©2015 Rocket Software, Inc. All Rights Reserved. License Management What changed? Is the license count wrong? Identify the source Logging
  19. 19. 19 Wrong Count? maxusers where (az • +, #, = d3 –qn (Unix) or %GetNetLicenseInfo() (Windows) ©2015 Rocket Software, Inc. All Rights Reserved.
  20. 20. 20 Identify the Source Is there a pattern? • Happens once a night? • Do a lot of kill -9 and/or reset-user? Types of connections? • ODBC • telnet • MVSP • RPC ©2015 Rocket Software, Inc. All Rights Reserved.
  21. 21. 21 Logging Use the –v option with the License Manager • Generates a log file D3 Unix logging • Use the –v option with parameter to turn on d3 logging • Generates several log files ©2015 Rocket Software, Inc. All Rights Reserved.
  22. 22. 22 Overflow Management
  23. 23. 23©2015 Rocket Software, Inc. All Rights Reserved. Overflow Management What changed? Commands Logs Recovery
  24. 24. 24 Overflow Management Is it happening now? Which process is doing it? What is the process doing to cause it? ©2015 Rocket Software, Inc. All Rights Reserved.
  25. 25. 25 Overflow Management - Commands ovf (at where dump <FID> ix ©2015 Rocket Software, Inc. All Rights Reserved. where 1 lz Ln PCB PIB ABS Stat R1 & Return stack contents 06 Oct 2015 FID Stat Base 11:15:11 *0001 00000081 FF90 000018 0 OVF_ADD:000 BR_STR:000
  26. 26. 26 Overflow Management – dump Command ©2015 Rocket Software, Inc. All Rights Reserved. dump 13709 ix fid: 13709 : 0 0 0 0 ( 358D : 0 0 0 0 ) ARNI 00 0001 00 026A27C8 4426 FM_CREATE_SUB:000 FM_DO_GDM:000 FM_CREATE_FILE:000 ARNI 00 0001 00 030FE871 4425 UPD_REL_DPTR:000 UPD_END:000 FM_DELETE_FILE:000 0000 0000001A 1300B001 74657374 FE333539 0 :........test^359: 0010 34FEFFFF FFFFFFFF FFFFFFFF FFFFFFFF 016 :4^______________: Most recent check-out: Signature, stat, line, level, time, date Most recent check-out: Return stack Most recent check-in: Signature, stat, line, level, time, date Most recent check-in: Return stack
  27. 27. 27 Overflow Management – Logs & Recovery Logs • list-errors • errors,ovf Recovery • Full restore • reclaim-ovf ©2015 Rocket Software, Inc. All Rights Reserved.
  28. 28. 28 Lockups (Hangs)
  29. 29. 29©2015 Rocket Software, Inc. All Rights Reserved. Lockups What changed? Not hung (just super slow)? Entire server? All of D3? Group of D3 users? A single D3 user?
  30. 30. 30 Lockups Not Hung? • where • strace (Unix) Entire Server? • Wait, reboot & analyze logs ©2015 Rocket Software, Inc. All Rights Reserved.
  31. 31. 31 Lockups – All of D3 (Windows) Generate a process dump • Task manager o Find the d3vme.exe o Right click on it o Click “Create dump file” o The location of the dump file will be displayed • sysinternal.com ProcDump utility o Must be downloaded and decompressed on system o procdump –ma d3vme.exe • Provide dumps for technical support if requested ©2015 Rocket Software, Inc. All Rights Reserved. __
  32. 32. 32 Lockups – All of D3 (Unix) Check semaphores • d3 –q • monitor debugger (s command) strace ©2015 Rocket Software, Inc. All Rights Reserved.
  33. 33. 33 D3 Lockups – Group of lines where Ln PCB PIB ABS Stat R1 & Return stack contents 09 Sep 2015 FID Stat Base 10:14:07 0000 00000080 F310 000018 AU_INPUT:000 AU_START_TCL:000 AU_TCL:000 TCL1:000 0001 00000081 F390 000018 AU_INPUT:000 AU_GET_CMND:000 0002 00000082 BF90 000018 LK_FAIL:000 RET_SRCH:000 BR_READITM:000 BR_READ:000 0003 00000083 BF90 000018 LK_FAIL:000 RET_SRCH:000 BR_READITM:000 BR_READ:000 *0004 00000084 FF90 000018 WS_WHERE1:000 WS_WHATWHERE:000 0256 00000180 BF10 000018 SP_SLEEP:000 SP_SPOOLOUT:000 0513 0000DBA7 BF10 000018 P PP_SCHED:000 list-lockq Waiting owner fcb group line# level line# level fid fid item id 0002 00 0001 00 00000A37 00000A28 who 0003 00 0001 00 00000A37 00000A28 who ©2015 Rocket Software, Inc. All Rights Reserved.
  34. 34. 34 D3 Lockups – A Single Line where 1 lz Ln PCB PIB ABS Stat R1 & Return stack contents 08 Oct 2015 FID Stat Base 13:25: 27 *0001 00000081 FF90 000018 0 WS_WHERE1:000 WS_WHATWHERE:000 pibstat ff90 PIB Status ff90 1111 1111 1001 0000 Process is running or activatable not sleeping not frame faulted echoing input not I/O roadblocked not reading character not writing character not comatized DCD on not received XOFF type-ahead enabled modem-control enabled Summary Status: Process is currently using the CPU ©2015 Rocket Software, Inc. All Rights Reserved.
  35. 35. 35 Crashes
  36. 36. 36©2015 Rocket Software, Inc. All Rights Reserved. Crashes What changed? Collecting information Catching it in the act
  37. 37. 37©2015 Rocket Software, Inc. All Rights Reserved. Crashes On time crash? Is there a pattern? • Everyday at 6PM • Right after virus scanner runs Type of connections to D3? Anything unusual before the crash?
  38. 38. 38©2015 Rocket Software, Inc. All Rights Reserved. Collecting Information Errors file Runtime-errors file Jobs file Trace files Windows Event Log/Unix errpt & perrpt d3watcher log files ProcDump/core dump
  39. 39. 39©2015 Rocket Software, Inc. All Rights Reserved. Catching It in the Act Install and configure d3watcher Enable tracing Download and enable ProcDump (Windows)
  40. 40. 40 Summary Drill down to provide as much information as possible Call Rocket Technical Support for assistance Be prepared to answer the question: What changed? ©2015 Rocket Software, Inc. All Rights Reserved.
  41. 41. 41 Disclaimer THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON ROCKET SOFTWARE’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY ROCKET SOFTWAREWITHOUT NOTICE. ROCKET SOFTWARE SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: • CREATING ANY WARRANTY OR REPRESENTATION FROM ROCKET SOFTWARE(OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR • ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF ROCKET SOFTWARE. ©2015 Rocket Software, Inc. All Rights Reserved.
  42. 42. 42 Trademarks and Acknowledgements The trademarks and service marks identified in the following list are the exclusive properties of Rocket Software, Inc. and its subsidiaries (collectively, “Rocket Software”). These marks are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. Not all trademarks owned by Rocket Software are listed. The absence of a mark from this page neither constitutes a waiver of any intellectual property rights that Rocket Software has established in its marks nor means that Rocket Software is not owner of any such marks. Aldon, CorVu, Dynamic Connect, D3, FlashConnect, Pick, mvBase, mvEnterprise, NetCure, Rocket, SystemBuilder, U2, U2 Web Development Environment, UniData, UniVerse, and wIntegrate Other company, product, and service names mentioned herein may be trademarks or service marks of others. ©2015 Rocket Software, Inc. All Rights Reserved.
  43. 43. 43

×