1.
Session ID:
Prepared by:
8 Advance Topics on Supporting
JDE
David Toal, Terillium

2.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Topics
• 100% CPU on JAS Servers
• Network Causing JDE to Crash
• Backup Strategy Crashing JD Edwards
• Database locking and blocking causing sporadic
performance issues?
• Largest Table in Database
• SQL Maintenance
• Index on a table improve performance?
• Add a second Web server to Production?

3.
100% CPU on Jas Servers

4.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Symptoms
• WebServer CPU is running at or
near 100%
• Users are experiencing slow
and non responsive behavior.
• Some Users might not be able
to log on.
• Other instances on same server
are slow as well.

5.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Multiple Causes = Same Resolution
Cause
• User Pulls Large amount of
Data. (DataBrowser, App)
• Custom Code Heavy Data
Calculations in ER.
• Standard Code Issues –
P30200, P90CG501 , P4310,
P04015, P40215…. (21033380,
2174901, 19995782)
• Tools Release Bugs (18787865,
18921015, 19711614)
• WebLogic Bugs (13836819,
23094342, 16857433…)
Resolution
• Restart the JAS Server

6.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Under the Covers – Garbage Collection
CPU is running 100%
because Java Garbage
Collection Running Non-
Stop.
Garbage Collections Result
in “Stop the World” events.
http://www.oracle.com/webfolder/technetwork/tutorials/obe/java/gc01/index.html

7.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Under the Cover – Infinite Loops
CPU is running 100%
because of a code infinite
loop.
Infinite loops use as much
CPU as possible, but
typically do not get in the
way of other requests.
If Condition
True
Run Code
Check
Condition
Unchanged
Set
Condition
to true

8.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Troubleshooting
• Read the Log files
• Jas
• Weblogic
• Potentially Java logging
• Identify the Situation
• What are users doing?
• Investigate Known Bugs and issue
• Apps, Tools, Weblogic and Java

9.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Resolutions
• Fix Custom Code
• Apply ESU’s
• Apply Tools release
• Update/Patch Weblogic
• Update Java

10.
Networking Causing JDE to
Crash

11.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Error Messages

12.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Physical Network

13.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
VMWare
https://pubs.vmware.com/vsphere-51/topic/com.vmware.vsphere.avail.doc/GUID-52F1BC6A-CC0D-4B1A-BDD7-5063B3AED1CE.html

14.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Resolution - Recommendations
• Find the Metadata Kernel and verify that the disconnect
is in its logs. Kill it. A new one will be generated.
• Find the Security Server Kernel with the disconnects and
kill it.
• All JDE Servers should be on the same VLAN, Subnet,
Switch(es).
• Traffic between servers should never be routed
• If Network has issue, Test JDE do not assume it is OK.
• Simple, Network can’t go down.

15.
Backup Crashing JDE

16.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Symptoms
• JDE is running on Virtual Hardware
• User’s can’t log into JDE first think in the morning
(especially Monday Morning), however not every
Monday.
• Log files look like a Network issue with lost connections.
• In Windows the Event Viewer looks clean except for a
Time Adjustment.
• Problem appears to happen at the end of the backups,
not the beginning or during

17.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Snapshot Backup Initialized
San
Server-1
C_Drive.File
D_Drive.File
Server-2
C_Drive.File
D_Drive.File
Server-3
C_Drive.File
D_Drive.File
Physical Server
Physical Server
Server-1
Server-2
Server-3
Back-
up
Delta
Network

18.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Snapshot Removed
San
Server-1
C_Drive.File
D_Drive.File
Server-2
C_Drive.File
D_Drive.File
Server-3
C_Drive.File
D_Drive.File
Physical Server
Physical Server
Server-1
Server-2
Server-3
Back-
up
Delta
Network

19.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Back to Normal Operations
San
Server-1
C_Drive.File
D_Drive.File
Server-2
C_Drive.File
D_Drive.File
Server-3
C_Drive.File
D_Drive.File
Physical Server
Physical Server
Server-1
Server-2
Server-3
Back-
up
Network

20.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Resolution
• Only do Backups during slow to no activity times
• Restart JD Edwards after Backups
• Use Database Tools to do Database Backups
• WebServers/BSSV/AIS Rarely Change – Limit Backups
• Enterprise Servers – PrintQueue, Packages change
regularly
• Deployment Server does not stability of production.

21.
Database Locking and Blocking

22.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Symptoms
• Endless Spinning Wheel
• Slow responses in some applications, but fine in others.
• Batch Job never finishing
• Batches getting backed up in queues

23.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
SQL Server – Activity Monitor

24.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Oracle Server
Script to Identify Blockers
• select * from dba_blockers
• will show the Oracle SID(s) of sessions blocking
You can then take that to v$session as in:
• select username, sid, serial#, program, machine, event,
seconds_in_wait from v$session where sid = NNN
• (NNN from first query)

25.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Resolutions
• All - Determine the Code the is causing the block and fix
it.
• All – Stop the offending process, Typically JDE call object
or Batch Job Kernel
• Oracle – Kill Session (alter system kill session ‘sid’,’serial’)
• SQL Server – Turn on RCSI and Snap Shot Isolation.
Which will allow dirty reads of the data
• SQL Server – Use NoLock Option in third party database
accsees
• Select * from PRODDTA.F0101 with (NOLOCK)

26.
Largest Table in Database

27.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Finding the Largest Tables
SQL Server
SELECT
t.NAME AS TableName,
s.Name AS SchemaName,
p.rows AS RowCounts,
SUM(a.total_pages) * 8 AS TotalSpaceKB,
SUM(a.used_pages) * 8 AS UsedSpaceKB,
(SUM(a.total_pages) - SUM(a.used_pages)) * 8 AS UnusedSpaceKB
FROM
sys.tables t
INNER JOIN
sys.indexes i ON t.OBJECT_ID = i.object_id
INNER JOIN
sys.partitions p ON i.object_id = p.OBJECT_ID AND i.index_id =
p.index_id
INNER JOIN
sys.allocation_units a ON p.partition_id = a.container_id
LEFT OUTER JOIN
sys.schemas s ON t.schema_id = s.schema_id
WHERE
t.NAME NOT LIKE 'dt%'
AND t.is_ms_shipped = 0
AND i.OBJECT_ID > 255
GROUP BY
t.Name, s.Name, p.Rows
ORDER BY
SUM(a.used_pages) desc
From <http://stackoverflow.com/questions/7892334/get-size-of-all-
tables-in-database>
Oracle
select * from (
select owner, segment_name, bytes/1024/1024 Size_Mb from
dba_segments order by bytes/1024/1024 DESC )
where rownum <= 20
From <http://www.freelists.org/post/oracle-l/Finding-top-20-large-
objectstables-in-database,2>

28.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Example
TableName SchemaName RowCounts
TotalSpace
KB
UsedSpace
KB
UnusedSpace
KB
F42199 PRODDTA 282934259 751738536 751620376 118160
F0911 PRODDTA 79698847 379647592 377779440 1868152
F0902 PRODDTA 26811210 69369864 69361976 7888
F0901 PRODDTA 27226149 63510512 63491096 19416
F550902F PRODDTA 228711645 41898536 41892344 6192
F0618 PRODDTA 7087448 31130528 31053112 77416
F01131M PRODDTA 61101026 25117072 25113328 3744
F4111 PRODDTA 8179761 23409368 23390768 18600
F0411 PRODDTA 3953236 12748568 12639984 108584

29.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Resolutions
• Run you JDE Purge Programs (F42199, F986110,
F0101Z1, etc.)
• Implement Archiving Solutions like Essentio and
Archtools
• Scheduled SQL Scripts to clear outdated data.

30.
SQL Maintenance

31.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
SQL Fragmentation
SELECT dbschemas.[name] as 'Schema',
dbtables.[name] as 'Table',
dbindexes.[name] as 'Index',
indexstats.avg_fragmentation_in_percent,
indexstats.page_count
FROM sys.dm_db_index_physical_stats (DB_ID(), NULL, NULL, NULL, NULL) AS indexstats
INNER JOIN sys.tables dbtables on dbtables.[object_id] = indexstats.[object_id]
INNER JOIN sys.schemas dbschemas on dbtables.[schema_id] = dbschemas.[schema_id]
INNER JOIN sys.indexes AS dbindexes ON dbindexes.[object_id] = indexstats.[object_id]
AND indexstats.index_id = dbindexes.index_id
WHERE indexstats.database_id = DB_ID()
ORDER BY indexstats.avg_fragmentation_in_percent desc

32.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Schema Table Index
avg_fragmentation_in_
percent page_count
SY920 F95631 NULL 99.9688958 51407
SVM920 F986110 NULL 99.83630952 53711
SY920 F95631 F95631_5 99.18525703 10310
SY920 F95631 F95631_4 99.00985317 20704
SVM920 F986110 F986110_4 98.67240761 22296
SY920 F98211 F98211_5 98.11715481 10516
OL920 F9861 F9861_PK 94.39737274 15225
SY920 F98211 F98211_PK 91.51252208 38492
SVM920 F986114 F986114_2 86.0855957 14697
SY920 F95630 NULL 76.28625873 37744
SY920 F98210 F98210_PK 73.26782145 24016
DD920 F00165 NULL 63.53686636 13860
SVM920 F986110 F986110_PK 55.73337359 13247
SVM920 F986114 F986114_PK 50.08852006 36150
SVM920 F986114A F986114A_PK 35.96706725 594302
DD920 F9210 F9210_PK 25.46419098 10556
SY920 F9563110 F9563110_PK 1.444407121 35724
SY920 F956311A F956311A_PK 0.539741028 77074
SVM920 F986110 NULL 0 574778
SY920 F95630 NULL 0 54295
SY920 F95631 NULL 0 710468

33.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Rebuild vs ReOganize
• Rebuild is always an offline
operation and has to finish in
order to fix indexing problems
• Reoganize can be done online
but only reorganizes may not
work well with highly
fragmented tables
avg_fragmentation_in_percent value Corrective statement
> 5% and < = 30% ALTER INDEX REORGANIZE
> 30% ALTER INDEX REBUILD WITH (ONLINE = ON)*

34.
Indexes

35.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Index added to F03B11Z1, F0411Z1,
F0911Z1
“I just tested several batches and the run time improved
significantly on the large batches. Batch 600000000819
and 600000000831 went from over one hour+ to less than
5 minute”
- Happy Customer

36.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Resolution
• Identify Indexes needed using Database Tools and
Developer expertise. (aka understand the code)
• Build the Index in JDE Edwards (you can do it at the
database level but you will loose it during an upgrade)
• Don’t over do it! Many time indexes will help but it is
possible to create an index that has a negative impact on
performance.

37.
Add a Second Production Web
Server

38.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Symptoms
• Best Practices says 50-100 users per Jas instance.
• Need to be able to able to bring a webserver down but
not all of JDE.
• Want to leave the Webservers up for testing but keep
users out.
• Want to use a Alias URL eg. jde.company.net

39.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Resolution
• Option 1 – Give different users different URL’s and
effectively load balance by picking who goes where
Web 1 Web 2

40.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Resolution
• Option 2 – Use DNS Round Robin and have users hit
different servers.
Web 1 Web 2

41.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Resolution
• Option 3 – Use a Load Balancer
Web 1 Web 2
Load Balancer

42.
April 2-6, 2017 in Las Vegas, NV USA #C17LV
Contact us: weknowjde@terillium.com
Questions?