Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

8 Advanced JDE Support Topics (That Are Often Overlooked)

3,039 views

Published on

Current situation: You’ve gone live on Oracle JD Edwards EnterpriseOne, recently having completed an upgrade or a new implementation, and now you’re noticing signs that something isn’t quite right. How do you determine what is going on with your system – where do you start trouble shooting?

Published in: Technology
  • Login to see the comments

8 Advanced JDE Support Topics (That Are Often Overlooked)

  1. 1. Session ID: Prepared by: 8 Advance Topics on Supporting JDE David Toal, Terillium
  2. 2. April 2-6, 2017 in Las Vegas, NV USA #C17LV Topics • 100% CPU on JAS Servers • Network Causing JDE to Crash • Backup Strategy Crashing JD Edwards • Database locking and blocking causing sporadic performance issues? • Largest Table in Database • SQL Maintenance • Index on a table improve performance? • Add a second Web server to Production?
  3. 3. 100% CPU on Jas Servers
  4. 4. April 2-6, 2017 in Las Vegas, NV USA #C17LV Symptoms • WebServer CPU is running at or near 100% • Users are experiencing slow and non responsive behavior. • Some Users might not be able to log on. • Other instances on same server are slow as well.
  5. 5. April 2-6, 2017 in Las Vegas, NV USA #C17LV Multiple Causes = Same Resolution Cause • User Pulls Large amount of Data. (DataBrowser, App) • Custom Code Heavy Data Calculations in ER. • Standard Code Issues – P30200, P90CG501 , P4310, P04015, P40215…. (21033380, 2174901, 19995782) • Tools Release Bugs (18787865, 18921015, 19711614) • WebLogic Bugs (13836819, 23094342, 16857433…) Resolution • Restart the JAS Server
  6. 6. April 2-6, 2017 in Las Vegas, NV USA #C17LV Under the Covers – Garbage Collection CPU is running 100% because Java Garbage Collection Running Non- Stop. Garbage Collections Result in “Stop the World” events. http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html
  7. 7. April 2-6, 2017 in Las Vegas, NV USA #C17LV Under the Cover – Infinite Loops CPU is running 100% because of a code infinite loop. Infinite loops use as much CPU as possible, but typically do not get in the way of other requests. If Condition True Run Code Check Condition Unchanged Set Condition to true
  8. 8. April 2-6, 2017 in Las Vegas, NV USA #C17LV Troubleshooting • Read the Log files • Jas • Weblogic • Potentially Java logging • Identify the Situation • What are users doing? • Investigate Known Bugs and issue • Apps, Tools, Weblogic and Java
  9. 9. April 2-6, 2017 in Las Vegas, NV USA #C17LV Resolutions • Fix Custom Code • Apply ESU’s • Apply Tools release • Update/Patch Weblogic • Update Java
  10. 10. Networking Causing JDE to Crash
  11. 11. April 2-6, 2017 in Las Vegas, NV USA #C17LV Error Messages
  12. 12. April 2-6, 2017 in Las Vegas, NV USA #C17LV Physical Network
  13. 13. April 2-6, 2017 in Las Vegas, NV USA #C17LV VMWare https://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.avail.doc/GUID-52F1BC6A-CC0D-4B1A-BDD7-5063B3AED1CE.html
  14. 14. April 2-6, 2017 in Las Vegas, NV USA #C17LV Resolution - Recommendations • Find the Metadata Kernel and verify that the disconnect is in its logs. Kill it. A new one will be generated. • Find the Security Server Kernel with the disconnects and kill it. • All JDE Servers should be on the same VLAN, Subnet, Switch(es). • Traffic between servers should never be routed • If Network has issue, Test JDE do not assume it is OK. • Simple, Network can’t go down.
  15. 15. Backup Crashing JDE
  16. 16. April 2-6, 2017 in Las Vegas, NV USA #C17LV Symptoms • JDE is running on Virtual Hardware • User’s can’t log into JDE first think in the morning (especially Monday Morning), however not every Monday. • Log files look like a Network issue with lost connections. • In Windows the Event Viewer looks clean except for a Time Adjustment. • Problem appears to happen at the end of the backups, not the beginning or during
  17. 17. April 2-6, 2017 in Las Vegas, NV USA #C17LV Snapshot Backup Initialized San Server-1 C_Drive.File D_Drive.File Server-2 C_Drive.File D_Drive.File Server-3 C_Drive.File D_Drive.File Physical Server Physical Server Server-1 Server-2 Server-3 Back- up Delta Network
  18. 18. April 2-6, 2017 in Las Vegas, NV USA #C17LV Snapshot Removed San Server-1 C_Drive.File D_Drive.File Server-2 C_Drive.File D_Drive.File Server-3 C_Drive.File D_Drive.File Physical Server Physical Server Server-1 Server-2 Server-3 Back- up Delta Network
  19. 19. April 2-6, 2017 in Las Vegas, NV USA #C17LV Back to Normal Operations San Server-1 C_Drive.File D_Drive.File Server-2 C_Drive.File D_Drive.File Server-3 C_Drive.File D_Drive.File Physical Server Physical Server Server-1 Server-2 Server-3 Back- up Network
  20. 20. April 2-6, 2017 in Las Vegas, NV USA #C17LV Resolution • Only do Backups during slow to no activity times • Restart JD Edwards after Backups • Use Database Tools to do Database Backups • WebServers/BSSV/AIS Rarely Change – Limit Backups • Enterprise Servers – PrintQueue, Packages change regularly • Deployment Server does not stability of production.
  21. 21. Database Locking and Blocking
  22. 22. April 2-6, 2017 in Las Vegas, NV USA #C17LV Symptoms • Endless Spinning Wheel • Slow responses in some applications, but fine in others. • Batch Job never finishing • Batches getting backed up in queues
  23. 23. April 2-6, 2017 in Las Vegas, NV USA #C17LV SQL Server – Activity Monitor
  24. 24. April 2-6, 2017 in Las Vegas, NV USA #C17LV Oracle Server Script to Identify Blockers • select * from dba_blockers • will show the Oracle SID(s) of sessions blocking You can then take that to v$session as in: • select username, sid, serial#, program, machine, event, seconds_in_wait from v$session where sid = NNN • (NNN from first query)
  25. 25. April 2-6, 2017 in Las Vegas, NV USA #C17LV Resolutions • All - Determine the Code the is causing the block and fix it. • All – Stop the offending process, Typically JDE call object or Batch Job Kernel • Oracle – Kill Session (alter system kill session ‘sid’,’serial’) • SQL Server – Turn on RCSI and Snap Shot Isolation. Which will allow dirty reads of the data • SQL Server – Use NoLock Option in third party database accsees • Select * from PRODDTA.F0101 with (NOLOCK)
  26. 26. Largest Table in Database
  27. 27. April 2-6, 2017 in Las Vegas, NV USA #C17LV Finding the Largest Tables SQL Server SELECT t.NAME AS TableName, s.Name AS SchemaName, p.rows AS RowCounts, SUM(a.total_pages) * 8 AS TotalSpaceKB, SUM(a.used_pages) * 8 AS UsedSpaceKB, (SUM(a.total_pages) - SUM(a.used_pages)) * 8 AS UnusedSpaceKB FROM sys.tables t INNER JOIN sys.indexes i ON t.OBJECT_ID = i.object_id INNER JOIN sys.partitions p ON i.object_id = p.OBJECT_ID AND i.index_id = p.index_id INNER JOIN sys.allocation_units a ON p.partition_id = a.container_id LEFT OUTER JOIN sys.schemas s ON t.schema_id = s.schema_id WHERE t.NAME NOT LIKE 'dt%' AND t.is_ms_shipped = 0 AND i.OBJECT_ID > 255 GROUP BY t.Name, s.Name, p.Rows ORDER BY SUM(a.used_pages) desc From <http://stackoverflow.com/questions/7892334/get-size-of-all- tables-in-database> Oracle select * from ( select owner, segment_name, bytes/1024/1024 Size_Mb from dba_segments order by bytes/1024/1024 DESC ) where rownum <= 20 From <http://www.freelists.org/post/oracle-l/Finding-top-20-large- objectstables-in-database,2>
  28. 28. April 2-6, 2017 in Las Vegas, NV USA #C17LV Example TableName SchemaName RowCounts TotalSpace KB UsedSpace KB UnusedSpace KB F42199 PRODDTA 282934259 751738536 751620376 118160 F0911 PRODDTA 79698847 379647592 377779440 1868152 F0902 PRODDTA 26811210 69369864 69361976 7888 F0901 PRODDTA 27226149 63510512 63491096 19416 F550902F PRODDTA 228711645 41898536 41892344 6192 F0618 PRODDTA 7087448 31130528 31053112 77416 F01131M PRODDTA 61101026 25117072 25113328 3744 F4111 PRODDTA 8179761 23409368 23390768 18600 F0411 PRODDTA 3953236 12748568 12639984 108584
  29. 29. April 2-6, 2017 in Las Vegas, NV USA #C17LV Resolutions • Run you JDE Purge Programs (F42199, F986110, F0101Z1, etc.) • Implement Archiving Solutions like Essentio and Archtools • Scheduled SQL Scripts to clear outdated data.
  30. 30. SQL Maintenance
  31. 31. April 2-6, 2017 in Las Vegas, NV USA #C17LV SQL Fragmentation SELECT dbschemas.[name] as 'Schema', dbtables.[name] as 'Table', dbindexes.[name] as 'Index', indexstats.avg_fragmentation_in_percent, indexstats.page_count FROM sys.dm_db_index_physical_stats (DB_ID(), NULL, NULL, NULL, NULL) AS indexstats INNER JOIN sys.tables dbtables on dbtables.[object_id] = indexstats.[object_id] INNER JOIN sys.schemas dbschemas on dbtables.[schema_id] = dbschemas.[schema_id] INNER JOIN sys.indexes AS dbindexes ON dbindexes.[object_id] = indexstats.[object_id] AND indexstats.index_id = dbindexes.index_id WHERE indexstats.database_id = DB_ID() ORDER BY indexstats.avg_fragmentation_in_percent desc
  32. 32. April 2-6, 2017 in Las Vegas, NV USA #C17LV Schema Table Index avg_fragmentation_in_ percent page_count SY920 F95631 NULL 99.9688958 51407 SVM920 F986110 NULL 99.83630952 53711 SY920 F95631 F95631_5 99.18525703 10310 SY920 F95631 F95631_4 99.00985317 20704 SVM920 F986110 F986110_4 98.67240761 22296 SY920 F98211 F98211_5 98.11715481 10516 OL920 F9861 F9861_PK 94.39737274 15225 SY920 F98211 F98211_PK 91.51252208 38492 SVM920 F986114 F986114_2 86.0855957 14697 SY920 F95630 NULL 76.28625873 37744 SY920 F98210 F98210_PK 73.26782145 24016 DD920 F00165 NULL 63.53686636 13860 SVM920 F986110 F986110_PK 55.73337359 13247 SVM920 F986114 F986114_PK 50.08852006 36150 SVM920 F986114A F986114A_PK 35.96706725 594302 DD920 F9210 F9210_PK 25.46419098 10556 SY920 F9563110 F9563110_PK 1.444407121 35724 SY920 F956311A F956311A_PK 0.539741028 77074 SVM920 F986110 NULL 0 574778 SY920 F95630 NULL 0 54295 SY920 F95631 NULL 0 710468
  33. 33. April 2-6, 2017 in Las Vegas, NV USA #C17LV Rebuild vs ReOganize • Rebuild is always an offline operation and has to finish in order to fix indexing problems • Reoganize can be done online but only reorganizes may not work well with highly fragmented tables avg_fragmentation_in_percent value Corrective statement > 5% and < = 30% ALTER INDEX REORGANIZE > 30% ALTER INDEX REBUILD WITH (ONLINE = ON)*
  34. 34. Indexes
  35. 35. April 2-6, 2017 in Las Vegas, NV USA #C17LV Index added to F03B11Z1, F0411Z1, F0911Z1 “I just tested several batches and the run time improved significantly on the large batches. Batch 600000000819 and 600000000831 went from over one hour+ to less than 5 minute” - Happy Customer
  36. 36. April 2-6, 2017 in Las Vegas, NV USA #C17LV Resolution • Identify Indexes needed using Database Tools and Developer expertise. (aka understand the code) • Build the Index in JDE Edwards (you can do it at the database level but you will loose it during an upgrade) • Don’t over do it! Many time indexes will help but it is possible to create an index that has a negative impact on performance.
  37. 37. Add a Second Production Web Server
  38. 38. April 2-6, 2017 in Las Vegas, NV USA #C17LV Symptoms • Best Practices says 50-100 users per Jas instance. • Need to be able to able to bring a webserver down but not all of JDE. • Want to leave the Webservers up for testing but keep users out. • Want to use a Alias URL eg. jde.company.net
  39. 39. April 2-6, 2017 in Las Vegas, NV USA #C17LV Resolution • Option 1 – Give different users different URL’s and effectively load balance by picking who goes where Web 1 Web 2
  40. 40. April 2-6, 2017 in Las Vegas, NV USA #C17LV Resolution • Option 2 – Use DNS Round Robin and have users hit different servers. Web 1 Web 2
  41. 41. April 2-6, 2017 in Las Vegas, NV USA #C17LV Resolution • Option 3 – Use a Load Balancer Web 1 Web 2 Load Balancer
  42. 42. April 2-6, 2017 in Las Vegas, NV USA #C17LV Contact us: weknowjde@terillium.com Questions?

×