The document provides biographical information about Mahesh Vallampati including his career history working in IT roles for various companies, education background, white papers, blog, LinkedIn group, and contact information. It also discusses various performance issues that can occur with databases and applications and emphasizes the importance of properly identifying the root cause before blaming individuals or components. The case studies describe specific examples of performance problems encountered and the methods used to diagnose and resolve the issues.

1. Mahesh Vallampati

2.  Mahesh Vallampati
◦ Career
 Director of Oracle Applications and Technologies at Coach
America, the Bus Company.
 Sales and Consulting at Hotsos and SmartDog
 Director of IT Database Services at Ceva Logistics
 Sales and Consulting at Oracle for 9 years
◦ Education
 Courses in Business at Houston Community College
 Master’s in EE from Texas A&M University
 Career Focus
◦ Used to be DBA
◦ Now Techno-Functional (Fechnical)

3.  White Papers
◦ http://www.slideshare.net/mvallamp
 Email: mvallamp@yahoo.com
 Blog: http://mvallamp.blogspot.com
 Linked in Group Leader: DBA Manager
 Oracle Alumni Admin for content: 5000
members

4.  Performance Issues
 Blame
 Blame The Business
 Blame the Functional Team
 Blame the Developer
 Blame the DBA
 Conclusions

6.  Performance Issues
◦ Described in vague terms
 The System is slow
 The disk is slow
 The network is slow
 The SQL sucks
 The server sucks
 The problem is in the Application server
 The database is slow
 The code is awful
◦ My favorite
 Slow as molasses

7.  Issue Approach
◦ Sound sophisticated but not really effective (My
opinion)
 The BCHR is low or high
 We need to run Stats-pack
 AWR is better
 No, ASH is better
 “My” method is better
 Have you run statistics?
 Have you rebuilt the indexes?
 Let us take a look at the top 10 SQL statements
 The average physical I/O is high

9.  Let us Blame somebody
 Let us not leave anybody out
 Let us Blame
◦ The Business People
◦ The Functional Team
◦ The Developers
◦ The DBA’s

11.  Manufacturing Company
 In the Midwest
 Migrated from Mainframe to Oracle
Applications (11.5.10)
 Did Big Bang implementation of
Manufacturing, Order Management and
Financials
 System was barely usable
 Flew there the day they went live after
receiving panic calls

12.  Implementation vendor was there
 Another “expert” vendor was there
◦ BCHR and Block Size changes were being discussed
 Every theory possible was floating around
 Listened for an hour to the issues
 Asked a basic question
◦ Are you losing money and if so why?

13.  $$$$
◦ We are losing money
◦ We can’t take orders
◦ Order Management is not able to take orders
 It is taking 10 minutes to enter a 6 line order
 It used to take less than a minute in the
mainframe
 Good place to start
 They can pay my bills if only they make
money

14.  Traced the Order Management process
◦ Entering a header
◦ Entering a line
 Line Trace
◦ Entering a line was taking 1 minute and a half
 Most of the time was spent on calculating the
price of an item
 Enter “Advanced” Pricing

15.  Query was 9 pages long
 Query executed 319,000 Logical I/O’s

16.  Why are you using advanced pricing?
 We can make more money
 Let us try Simple Pricing
 Turned off Advanced Pricing
 6 line Order with Simple Pricing took 1
minute
 Interesting!!!

17.  Advanced Pricing
 Parameters for determining price
◦ Checks Overall credit Limit (AR)
◦ Checks outstanding balances (AR)
◦ Checks inventory levels (Inv)
◦ Checks overall order totals (Current Order)
◦ Checks payment history (AR)

18.  Can we use Simple Pricing instead of
Advanced Pricing?
 Enter the “Business”
 Discussion with COO
◦ Why are you using Advanced Pricing?
 We can make more money.
◦ How much more money can you make with
Advanced Pricing instead of Simple Pricing?
 Don’t know

19.  Potential Upside on a 1billion USD in revenue
◦ 4 million a year if we are lucky
 Investment required to make the system
perform at acceptable levels
◦ Cost – 10 to 15 million dollars
 Standby Reporting
 Archive/Purge
 Hardware Investments
 Software Investments
 Investments in people, processes etc.

20. Probability 50%
Discount Rate 5%
Year1 Year2 Year3 Year4 Year5
Revenue
Upside $4,000,000 $4,000,000 $4,000,000 $4,000,000 $4,000,000
NPV $17,317,907 NPV for 5 Years
Probable
Revenue $8,658,953 NPV times the Probability
Investment
Required $15,000,000 Capital Investment Required. Depreciation not included.
Profit ($6,341,047) Revenue-Cost Incurred

21.  Question to the COO
◦ Would you spend 15 million USD in today’s dollars
for a potential upside of 4 million dollars
◦ Economic forecast is gloomy
◦ Interest and Borrowing costs are high
 Right Answer, No.

22.  Changed from Advanced Pricing to Simple
Pricing
 The system was fast in performing
transactions
 System had over-capacity
 Even in month end, the system did not even
flinch
 Performance team disbanded
 Rebuilding Indexes and tables into bigger
block table-spaces got a yawn
 Solved in 2 days

23.  There is not enough hardware in the world to
support an irrational business requirement
 Ask why complex functionality is required when
simple functionality will do
 Ask the “Business” to justify need for “Advanced
Functionality”
 Solution
◦ Switch to Simple Pricing
◦ System Stabilized
◦ Project started on Monday Morning
◦ Project Ended Wednesday Afternoon
◦ Back Home Wednesday night

25.  Manufacturing Company in Ohio
 Migrated to Oracle Applications
 Severe Performance Issues
 Order Management is slow
 Call center for customers is just horrific
 Each call taking 10-15 minutes when before
each call during the mainframe was taking 2-
3 minutes
 Customer was using Configurator and BOM

26.  Method-R
 Traced the Order Management Process which
uses the Configurator
 Identified the SQL Statement that was taking
a long time

27.  SQL statement was performing close to
900,000 Logical I/O’s per execution
 SQL Statement was getting executed for every
item in the BOM that makes up an order line
 The more complex the BOM, the more items
in the BOM, the slower the process
 The SQL Statement was determining whether
the item needed to be displayed in the
invoice for the customer and in the packing
slip

28.  1. SELECT 'x'
 2. FROM BOM_BILL_OF_MATERIALS BOM, BOM_INVENTORY_COMPONENTS COMP,
 3. MTL_SYSTEM_ITEMS_B MTL, MTL_SYSTEM_ITEMS_B MTL_BOM, MTL_PARAMETERS
MP
 4. WHERE BOM.COMMON_BILL_SEQUENCE_ID = COMP.BILL_SEQUENCE_ID
 5. AND COMP.COMPONENT_ITEM_ID = MTL.INVENTORY_ITEM_ID
 6. AND MTL.ORGANIZATION_ID = MP.ORGANIZATION_ID
 7. AND (MTL_BOM.SEGMENT1 LIKE '%_ITEMS'
 8. OR MTL_BOM.SEGMENT1 LIKE '%_ITEM'
 9. OR MTL_BOM.SEGMENT1 LIKE '%_ITEMS2'
 10. OR MTL.BOM_ITEM_TYPE = :B3
 11. OR MTL.SEGMENT1 LIKE 'OC_%')
 12. AND BOM.ASSEMBLY_ITEM_ID = MTL_BOM.INVENTORY_ITEM_ID
 13. AND BOM.ORGANIZATION_ID = MP.ORGANIZATION_ID
 14. AND BOM.ORGANIZATION_ID = MTL_BOM.ORGANIZATION_ID
 15. AND MP.ORGANIZATION_CODE = :B2
 16. AND NVL(COMP.EFFECTIVITY_DATE, SYSDATE) <= SYSDATE
 17. AND NVL(COMP.DISABLE_DATE, SYSDATE) >= SYSDATE
 18. AND COMP.COMPONENT_ITEM_ID = :B1

29.  Should we tune the SQL?
 A couple of Predicates were using leading
wildcards
◦ AND (MTL_BOM.SEGMENT1 LIKE '%_ITEMS'
◦ OR MTL_BOM.SEGMENT1 LIKE '%_ITEM'
◦ OR MTL_BOM.SEGMENT1 LIKE '%_ITEMS2'
◦ OR MTL.BOM_ITEM_TYPE = :B3
◦ OR MTL.SEGMENT1 LIKE 'OC_%')

30.  First instinct
◦ Blame the developer
 Met with the developer
◦ He said that was the functional spec
 Met with the functional person
◦ Asked them why they were looking for
 %ITEM
 %ITEMS
 %ITEMS2
 OC_%

31.  Functional Person explanation
◦ Oh, it is a convention we use
◦ If the item type ends in ITEM, ITEMS or ITEMS2 or
starts with OC, then display it
◦ Otherwise, don’t display it
◦ They were proud of it too

32.  Should we create a reverse string functional
index and switch the predicates?
 Umm..No

33.  Called a meeting
◦ Business Users
◦ Functional Team
◦ Developers
◦ DBA’s
 Took a dictionary to the meeting
◦ Asked them to find words that ended with M or S
quickly
◦ Everybody was looking as if I was not sane
 I explained that was what they were asking
the system

34.  Created a flexfield with a Y or N at the
inventory item level
 Updated the field to indicate the status
 Rewrote the SQL Statement to use the flag
instead of the convention
 SQL now used 40 Logical I/O’s
 System speed was so fast that the
productivity of the call center was up.

35.  Try to answer the SQL statement yourself as if
you did not have a system
 Would it make sense?
 If no, then the system is probably going to be
bad at it too and will be slow
 Systems can process and automate irrational
requests
 The cost and pain is going to be high
 System can do dumb but slow (Exadata
anybody?)

37.  Manufacturing Company in Illinois
 Customer had implemented Oracle
Applications
 Shipping Application is slow
 SR with Oracle Support going nowhere
 Cannot determine if it is custom or standard

38.  Method-R
 Traced the Shipping Process
 Each Line in the shipping screen was taking
20-25 seconds
 Identified the SQL Statement that was taking
a long time
 Each execution was taking 650,000 Logical
I/O’s.

40. SELECT fds.short_text
FROM wsh_delivery_assignments wda,
wsh_delivery_details wsh,
fnd_attached_docs_form_vl fad,
fnd_documents_short_text fds
WHERE wsh.released_status = 'Y'
AND wsh.delivery_detail_id = wda.delivery_detail_id
AND wda.delivery_id = 124901
AND wsh.source_code = 'OE'
AND fad.datatype_name = 'Short Text'
AND fad.usage_type = 'O'
AND fad.function_name = 'OEXOEORD'
AND fad.entity_name = 'OE_ORDER_HEADERS'
AND fds.media_id = fad.media_id
AND fad.pk1_value = wsh.source_header_id
AND ROWNUM = 1

41.  1 - filter(ROWNUM=1)
 2 - filter("WDA"."DELIVERY_ID"=124901)
 13 - access("DET"."DATA_OBJECT_CODE"='OE_ORDER_HEADERS' AND
"DET"."LANGUAGE"=USERENV('LANG'))
 15 - access("AF"."FUNCTION_NAME"='OEXOEORD')
 16 - filter("DCU"."ENABLED_FLAG"='Y')
 17 - access("DCU"."ATTACHMENT_FUNCTION_ID"="AF"."ATTACHMENT_FUNCTION_ID")
 18 - filter("D"."USAGE_TYPE"='O' AND "DCU"."CATEGORY_ID"="D"."CATEGORY_ID")
 19 - access("D"."DATATYPE_ID"="DAT"."DATATYPE_ID" AND "DAT"."LANGUAGE"=USERENV('LANG')
AND
 "DAT"."USER_NAME"='Short Text')
 20 - access("DCT"."CATEGORY_ID"="D"."CATEGORY_ID" AND "DCT"."LANGUAGE"=USERENV('LANG'))
 22 - access("DT"."DOCUMENT_ID"="D"."DOCUMENT_ID" AND "DT"."LANGUAGE"=USERENV('LANG'))
 24 - access("FDS"."MEDIA_ID"="DT"."MEDIA_ID")
 25 - filter("AD"."ENTITY_NAME"='OE_ORDER_HEADERS')
 26 - access("D"."DOCUMENT_ID"="AD"."DOCUMENT_ID")
 27 - filter("WSH"."RELEASED_STATUS"='Y' AND "WSH"."SOURCE_CODE"='OE')
 28 - access("WSH"."SOURCE_HEADER_ID"=TO_NUMBER("AD"."PK1_VALUE"))
 29 - access("WSH"."DELIVERY_DETAIL_ID"="WDA"."DELIVERY_DETAIL_ID")

42.  Oracle Financials has flexfields which have
multiple purposes
 They have to be varchar2
 It is a common practice to store key values in
these columns
 Key values tend to be numbers
 Joins are made to flexfields from these
keyfield
 If you use an implicit type conversion
happening in the plan, this is usually the
issue

43. SELECT fds.short_text
FROM wsh_delivery_assignments wda,
wsh_delivery_details wsh,
fnd_attached_docs_form_vl fad,
fnd_documents_short_text fds
WHERE wsh.released_status = 'Y'
AND wsh.delivery_detail_id = wda.delivery_detail_id
AND wda.delivery_id = 1555559
AND wsh.source_code = 'OE'
AND fad.datatype_name = 'Short Text'
AND fad.usage_type = 'O'
AND fad.function_name = 'OEXOEORD'
AND fad.entity_name = 'OE_ORDER_HEADERS'
AND fds.media_id = fad.media_id
AND fad.pk1_value = to_char(wsh.source_header_id)
AND ROWNUM = 1

44.  Query now executes 700 Logical I/O’s down
from 650,000 LIO’s
 Query now has sub-second response
 Productivity is now way up.
 Life is good

45.  Known he was joining a varchar to a number
field
 Recognize that it could be an issue
 Checked the number of Logical I/O’s it was
doing
 Examined the explain plan
 Can we have a “lint” for SQL?

47.  Customer was a food distributor in Illinois
 They were going live on Oracle Financials
 They had several customizations which they
were rolling out
 Customizations were released as a series of
scripts
 The scripts were rolled out at 2:30 PM
 Testing would start

48.  Performance was horrible
 Find Customers, Find Orders and Find
Shipment forms would take forever
 The next day the system performance was
fine
 Everything was fast
 Everybody were stunned
◦ Client DBA team
◦ Oracle Consulting Team

49.  Traced the processes using 10046
 A lot of joins to custom temporary tables
were observed for queries with poor
performance
 A little bit more digging and almost all
performance issues were related to the
temporary tables
 So what is that with these temporary tables
that performed well the day after they were
deployed

50.  After a few hours of searching, did a quick
explain plan in SQL Developer
 Noticed the “Rule” in the explain plan
 Wait a minute – we should be using CBO.
 This is the 9i database
 Hmm..Let us look at the temporary tables
 Aaah..ah

51.  Stats on temporary tables were not being
collected as a part of the code deployment
 A nightly process that gathered stats on all
objects that did not have statistics gathered
the rest
 That explained why performance was poor
the day the code was deployed and better the
day after

52.  This was interesting
 Whose job was it to have the stats gathering
script as a part of the deployment
 Was it the DBA?
 Or the developer?
 It is a moot point when IT is on the line
 Is it the linebacker or the safety’s fault in a
busted play?
 Who cares if you lose?

54.  Use Method-R to get to the root cause
 Use other methods to confirm the problem
◦ Statspack
◦ Grid Control
◦ Homegrown scripts
 Avoid your first instinct

55.  First Instinct
◦ Try to tune the SQL
◦ Try to solve the problem
◦ Wrong Approach
 Correct Approach
◦ Identify blame
◦ Assign Blame (Important)
◦ Ask the “makes sense” question
◦ Will it make us money?
◦ If not, don’t do it

56.  Eliminate the problem
 Train the user not to create the
problem
 Reschedule the problem
 Tune/Optimize the problem
 Relocate the problem