Following our EBS R12.1.3 upgrade, we experienced inconsistent runtime and resource utilization with the Accrual Reconciliation Load program. During one month end close, a plant accountant could run accrual reconciliation for three years of data in under 10 hours. The next month, the same amount of data would take upwards of 30 hours to load. This behavior made it hard to plan for month end closure; successful accrual load being crucial to the finals for the month. Through multiple rounds of testing, we were able to complete the reconciliation process with nearly 15 years of data in just 1 hour and 45 minutes.

1. Session ID:
Prepared by:
Taming Oracle EBS R12.x
Accrual Reconciliation Load
Reduce Accrual Reconciliation Load
Program Runtime Post R12.x
Upgrade from EBS 11.5.10.2
10161
Joshua Johnson
Sr. Oracle DBA
Simmons Prepared Foods Inc.
@jj_dba

2. Abstract
Following our EBS R12.1.3 upgrade, we experienced
inconsistent runtime and resource utilization with the Accrual
Reconciliation Load program. During one month end close, a
plant accountant could run accrual reconciliation for three years
of data in under 10 hours. The next month, the same amount of
data would take upwards of 30 hours to load.
This behavior made it hard to plan for month end closure;
successful accrual load being crucial to the finals for the month.
Through multiple rounds of testing, we were able to complete
the reconciliation process with nearly 15 years of data in just 1
hour and 45 minutes.
2

3. Taming Oracle EBS R12.x Accrual
Reconciliation Load
Who are we?

4. Presenters
■ Joshua Johnson
▪ Sr. Oracle Apps DBA
— Simmons Prepared Foods, Inc.
■ Ned French
▪ Senior Manager
— Centroid
4

5. 5
• Owned/Managed by Founding Partners Since 1997
• Centroid is one of Oracle’s Top 25 Strategic Partners/Platinum Partner Status
• National technology firm focused on Consulting, Managed Services, Cloud
Services and Resell of Oracle products that span the entire enterprise.
• From Applications to Technology to Infrastructure; Centroid Data Center 100%
Oracle Products: Apps, Tech, HW
• AWARDED: 2014, 2015 Oracle Specialized Partner of the
Year – North America; Engineered Systems
Booth #815

6. Agenda
■ Brief background on our Oracle EBS upgrade from 11i to R12
▪ Including hardware outline
■ OEM 12c Implementation
■ Server OS Level Modifications – Oracle Linux
▪ Huge Pages, Memory Limits
■ Oracle Database Modifications
▪ Indexes, PGA & SGA
■ Results – Test
■ Developing Our Plan of Attack
■ Maintenance Round 1
■ Maintenance Round 2
■ Results – Production
■ Current State
■ References
6

7. Our Oracle EBS Upgrade From 11i to R12
Upgrade from version 11.5.10 to 12.1.3

8. 11.5.10 Upgrade to R12.1.3
▪ Database Server
— OL 5.5 64-bit
— 8 Older non-HyperThreading
CPUs
— 128 GB memory
8
■ Upgrade completed using legacy Dell blade servers
▪ Application Server
— OL 5.5 32-bit
— 8 Older non-
HyperThreading CPUs
— 32 GB memory

9. 11.5.10 Upgrade to R12.1.3
Today marks 1010 days since Go Live!
9
Task Date Time
Upgrade Start 03.JUL.2013 18:00
11i Pre-upgrade Functional Setup Complete 03.JUL.2013 23:59
12.1.1 Patching Complete 04.JUL.2013 22:00
12.1.3 Patching Complete 05.JUL.2013 04:00
Post Upgrade Patching Complete 05.JUL.2013 06:00
Business Owner Testing Complete 06.JUL.2013 17:00
All EBS User Accounts Reactivated 07.JUL.2013 17:00

10. OEM 12c Implementation
Central repository for monitoring Oracle database
instances

11. Painting a Rather Bleak Picture - DB Console
11
Create Accounting 06-JULY-2013

12. Painting a Rather Bleak Picture - DB Console
12
What happened here?

13. OEM 12c Implementation
■ Dbconsole
▪ Not meant to monitor a
complex enterprise level
production environment.
▪ Some servers are not
currently configured to
monitor and log errors.
▪ Unable to create the in-
depth alerts necessary to
proactively monitor an
enterprise system.
■ OEM 12c
▪ Decreased time spent in
reactionary processes
▪ Increased Oracle
availability
▪ Improved system health
and reporting
▪ Increased efficiency in
generating reports for
Oracle during an SR
▪ Increased ability to provide
workload reports for our
Application Developers and
management staff.
13

14. Server OS Level Modifications –
Oracle Linux
• HugePages
• Memory Limits

15. Server OS Level Modifications – Oracle
Linux
■ Provide reserved, non-swappable
memory space for our SGA
▪ 192 GB HugePages
▪ 60 GB PGA Target
▪ 100 GB SGA Target
■ Based on recommendations from
the Oracle nmr.hugepages script.
HugePages
■ Hard and soft memlock set
to match 192 GB huge
pages
▪ 201,326,592 kilobytes
Memory Limits
15
$ grep -i huge /proc/meminfo
HugePages_Total: 98304
HugePages_Free: 48276
HugePages_Rsvd: 1173
HugePages_Surp: 0
Hugepagesize: 2048 kB
http://docs.oracle.com/cd/E37670_01/E37355/html/ol_config_hugepages.html
/etc/security/limits.conf
vm.nr_hugepages = 98304
/etc/sysctl.conf
soft memlock 201326592
hard memlock 201326592

16. Server OS Level Modifications – Oracle
Enterprise Linux
16http://docs.oracle.com/cd/E37670_01/E37355/html/ol_config_hugepages.html

17. Oracle Database Modifications
• Indexes
• PGA
• SGA

18. Oracle Database Modifications
Indexes Added to BOM, PO & XLA
18
■ BOM.CST_WRITE_OFF_DETAILS_N2
■ PO.PO_ACCRUAL_WRITE_OFFS_N5
■ XLA.XLA_AE_LINES_N99
■ XLA.XLA_AE_LINES_N5

19. Oracle Database Modifications
Dropped Partitioned XLA Indexes
19
■ XLA.XLA_AE_HEADERS_N1;
■ XLA.XLA_AE_HEADERS_N2;
■ XLA.XLA_AE_HEADERS_N3;
■ XLA.XLA_AE_HEADERS_N4;
■ XLA.XLA_AE_HEADERS_N5;
■ XLA.XLA_AE_HEADERS_N8;
■ XLA.XLA_AE_HEADERS_U1;
■ XLA.XLA_AE_LINES_N1;
■ XLA.XLA_AE_LINES_N4;
■ XLA.XLA_AE_LINES_U1;
■ XLA.XLA_DISTRIBUTION_LINKS_N1;
■ XLA.XLA_DISTRIBUTION_LINKS_N3;
■ XLA.XLA_DISTRIBUTION_LINKS_U1;
■ XLA.XLA_EVENTS_N1;
■ XLA.XLA_EVENTS_N2;
■ XLA.XLA_EVENTS_N3;
■ XLA.XLA_EVENTS_U1;
■ XLA.XLA_EVENTS_U2;
■ XLA.XLA_TRANSACTION_ENTITIES_N1;
■ XLA.XLA_TRANSACTION_ENTITIES_U1;

20. Oracle Database Modifications
Recreated Partitioned XLA Indexes
20

21. Oracle Database Modifications
Recreate script altered after encountering errors with the
newsort feature in 11gR2 in testing
21

22. Oracle Database Modifications
■ SGA increased from 42 GB to 100 GB
▪ SGA and PGA configured to fit within
the non-swappable hugepages.
SGA Settings
22
pga_aggregate_target=60g
sga_target=100g

23. Developing Our Plan of Attack
• Maintenance Round 1
• Maintenance Round 2

24. Developing Our Plan of Attack
■ Not including OS and Database updates
▪ 17 hours required for maintenance in test.
■ We began testing options to separate these tasks into
two smaller maintenance windows with the least
amount of system downtime.
How do we address the issue with the amount of
time necessary to complete all of these
maintenance items?
24

25. Maintenance Round 1
25
■ Applied patch 15836954_R12.FND.B
■ Dropped partitioned indexes for XLA
■ Recreated partitioned indexes for XLA
■ Created new indexes for BOM, PO & XLA
■ Gather Schema Statistics for XLA, BOM, PO, AR and
AP at 80% or better
■ Gather Table Statistics on XLA partitioned objects
■ Gather Table Stats on AP, PO, XLA

26. Maintenance Round 2
26
■ Applied patches:
■ 10194427:R12.XLA.B
■ 20108234:R12.BOM.C
■ 14129023:R12.XLA.B
■ Gather Schema Statistics for XLA, BOM, PO, AR and
AP at 80% or better
■ We restarted all EBS services after the patches applied
successfully.

27. Results – Test
• Test Database Hardware
• Cisco Blade – 8 CPUs & 384 GB memory

28. Results – Test
28
Process Timing
Apply Patch 15836954_R12.FND.B 45 minutes
Drop XLA partitioned Indexes 2 minutes
Recreate XLA partitioned Indexes 464 minutes
Create New Indexes AP, PO & XLA 37 minutes
Gather Schema Statistics for XLA, BOM, PO, AR, AP at
80% or better
120 minutes
Gather Table Statistics on XLA partitioned objects Failed!
Gather Table Stats on AP, PO, XLA Failed!

29. Results – Test
Errors Encountered
29
Error
Unable to recreate XLA_AE_LINES_N4 index manually or via the xdf script.
Error from xdf run:
ORA-17410 error occured during Index creation.
Manually run:
Error report -
SQL Error: No more data to read from socket
Solution
We created a modified script to create the indexes manually with the
newsort functionality disabled.
"_newsort_enabled"=false;
■ Confirmed that there is sufficient space in the APPS_TS_TX_IDX
tablespace prior to testing this solution.

30. Results – Production
• Production Hardware
• Dell Blade – 8 CPUs & 128 GB memory

31. Results – Production
31
Process Timing
Apply patch 15836954_R12.FND.B 35 minutes
Drop XLA partitioned Indexes 31 seconds
Recreate XLA partitioned Indexes 271 minutes
Create New Indexes AP, PO & XLA 33 minutes
Gather Schema Statistics for XLA, BOM, PO, AR and AP at
80% or better.
Gather Table Statistics on XLA partitioned objects.
Gather Table Stats on AP, PO and XLA.
These steps were combined into one SQL script.
733 minutes
Applied patches 10194427:R12.XLA.B, 20108234:R12.BOM.C
& 14129023:R12.XLA.B merged.
31 minutes
Gather Schema Statistics for XLA, BOM, PO, AR and AP at
80% or better.
360 minutes

32. Production – Current State
• New Production Hardware
• Cisco Blade – 16 CPUs & 384 GB memory

33. Multiple Create Accounting Executions:
4-JAN-2016
33
■ Multiple Create Accounting sessions for separate business
units and their pre-processors were run in draft mode the
morning of January 4, 2016.
■ At this time we were still enforcing the program
incompatibilities.
■ They were also able to run Accrual Reconciliation Load
without seeing an increase in system waits. In this system
we are able to run Accrual Reconciliation Load for 15+
years of data in under 2 hours

34. 34
Multiple Create Accounting Executions:
4-JAN-2016

35. References

36. References
■ Oracle Database Administrator's Reference 11g Release 2 for Linux and
UNIX-Based Operating Systems
■ E10839-23
■ Oracle Database Performance Tuning Guide 11g Release 2
■ E41573-03
■ Oracle Database PL/SQL Packages and Types Reference 11g Release 2
■ E40758-03
■ Oracle Financials Implementation Guide Release 12.1
■ E13425-05
■ Oracle Linux Administrator's Solutions Guide for Release 6
■ E37355-41 - January 2015
■ Oracle Subledger Accounting Implementation Guide Release 12.1
■ E13628-04
■ Accrual Reconciliation Load Run Has Slow Performance (Doc ID
1490578.1)
36

37. Questions?
37

38. Thank You
38
You may complete the session evaluation either
on paper or online via the mobile app
Please complete the session evaluation
We appreciate your feedback and insight
Session ID: 10161