Your SlideShare is downloading. ×
Mahesh Vallampati
   Mahesh Vallampati    ◦ Career      Director of Oracle Applications and Technologies at Coach       America, the Bus C...
   White Papers    ◦ http://www.slideshare.net/mvallamp   Email: mvallamp@yahoo.com   Blog: http://mvallamp.blogspot.co...
   Performance Issues   Blame   Blame The Business   Blame the Functional Team   Blame the Developer   Blame the DBA...
   Performance Issues    ◦ Described in vague terms        The   System is slow        The   disk is slow        The  ...
   Issue Approach    ◦ Sound sophisticated but not really effective (My      opinion)        The BCHR is low or high    ...
   Let us Blame somebody   Let us not leave anybody out   Let us Blame    ◦   The   Business People    ◦   The   Functi...
   Manufacturing Company   In the Midwest   Migrated from Mainframe to Oracle    Applications (11.5.10)   Did Big Bang...
   Implementation vendor was there   Another “expert” vendor was there    ◦ BCHR and Block Size changes were being discu...
   $$$$    ◦ We are losing money    ◦ We can’t take orders    ◦ Order Management is not able to take orders   It is taki...
   Traced the Order Management process    ◦ Entering a header    ◦ Entering a line   Line Trace    ◦ Entering a line was...
   Query was 9 pages long   Query executed 319,000 Logical I/O’s
   Why are you using advanced pricing?   We can make more money   Let us try Simple Pricing   Turned off Advanced Pric...
   Advanced Pricing   Parameters for determining price    ◦   Checks   Overall credit Limit (AR)    ◦   Checks   outstan...
   Can we use Simple Pricing instead of    Advanced Pricing?   Enter the “Business”   Discussion with COO    ◦ Why are ...
   Potential Upside on a 1billion USD in revenue    ◦ 4 million a year if we are lucky   Investment required to make the...
Probability              50%             Discount Rate              5%                             Year1           Year2  ...
   Question to the COO    ◦ Would you spend 15 million USD in today’s dollars      for a potential upside of 4 million do...
   Changed from Advanced Pricing to Simple    Pricing   The system was fast in performing    transactions   System had ...
   There is not enough hardware in the world to    support an irrational business requirement   Ask why complex function...
   Manufacturing Company in Ohio   Migrated to Oracle Applications   Severe Performance Issues   Order Management is s...
   Method-R   Traced the Order Management Process which    uses the Configurator   Identified the SQL Statement that wa...
   SQL statement was performing close to    900,000 Logical I/O’s per execution   SQL Statement was getting executed for...
   1.    SELECT x   2.    FROM BOM_BILL_OF_MATERIALS BOM, BOM_INVENTORY_COMPONENTS COMP,   3.    MTL_SYSTEM_ITEMS_B MTL...
   Should we tune the SQL?   A couple of Predicates were using leading    wildcards    ◦   AND (MTL_BOM.SEGMENT1 LIKE %_...
   First instinct    ◦ Blame the developer   Met with the developer    ◦ He said that was the functional spec   Met wit...
   Functional Person explanation    ◦ Oh, it is a convention we use    ◦ If the item type ends in ITEM, ITEMS or ITEMS2 o...
   Should we create a reverse string functional    index and switch the predicates?   Umm..No
   Called a meeting    ◦   Business Users    ◦   Functional Team    ◦   Developers    ◦   DBA’s   Took a dictionary to t...
   Created a flexfield with a Y or N at the    inventory item level   Updated the field to indicate the status   Rewrot...
   Try to answer the SQL statement yourself as if    you did not have a system   Would it make sense?   If no, then the...
   Manufacturing Company in Illinois   Customer had implemented Oracle    Applications   Shipping Application is slow ...
   Method-R   Traced the Shipping Process   Each Line in the shipping screen was taking    20-25 seconds   Identified ...
SELECT fds.short_textFROM wsh_delivery_assignments wda,wsh_delivery_details wsh,fnd_attached_docs_form_vl fad,fnd_document...
   1 - filter(ROWNUM=1)    2 - filter("WDA"."DELIVERY_ID"=124901)    13 - access("DET"."DATA_OBJECT_CODE"=OE_ORDER_HEAD...
   Oracle Financials has flexfields which have    multiple purposes   They have to be varchar2   It is a common practic...
SELECT fds.short_textFROM wsh_delivery_assignments wda,wsh_delivery_details wsh,fnd_attached_docs_form_vl fad,fnd_document...
   Query now executes 700 Logical I/O’s down    from 650,000 LIO’s   Query now has sub-second response   Productivity i...
   Known he was joining a varchar to a number    field   Recognize that it could be an issue   Checked the number of Lo...
   Customer was a food distributor in Illinois   They were going live on Oracle Financials   They had several customiza...
   Performance was horrible   Find Customers, Find Orders and Find    Shipment forms would take forever   The next day ...
   Traced the processes using 10046   A lot of joins to custom temporary tables    were observed for queries with poor  ...
   After a few hours of searching, did a quick    explain plan in SQL Developer   Noticed the “Rule” in the explain plan...
   Stats on temporary tables were not being    collected as a part of the code deployment   A nightly process that gathe...
   This was interesting   Whose job was it to have the stats gathering    script as a part of the deployment   Was it t...
   Use Method-R to get to the root cause   Use other methods to confirm the problem    ◦ Statspack    ◦ Grid Control    ...
   First Instinct    ◦ Try to tune the SQL    ◦ Try to solve the problem    ◦ Wrong Approach   Correct Approach    ◦   I...
 Eliminate the problem Train the user not to create the  problem Reschedule the problem Tune/Optimize the problem Rel...
Enough Blame for System Performance Issues
Enough Blame for System Performance Issues
Enough Blame for System Performance Issues
Enough Blame for System Performance Issues
Enough Blame for System Performance Issues
Enough Blame for System Performance Issues
Enough Blame for System Performance Issues
Enough Blame for System Performance Issues
Enough Blame for System Performance Issues
Upcoming SlideShare
Loading in...5
×

Enough Blame for System Performance Issues

836

Published on

Published in: Technology
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total Views
836
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Enough Blame for System Performance Issues"

  1. 1. Mahesh Vallampati
  2. 2.  Mahesh Vallampati ◦ Career  Director of Oracle Applications and Technologies at Coach America, the Bus Company.  Sales and Consulting at Hotsos and SmartDog  Director of IT Database Services at Ceva Logistics  Sales and Consulting at Oracle for 9 years ◦ Education  Courses in Business at Houston Community College  Master’s in EE from Texas A&M University Career Focus ◦ Used to be DBA ◦ Now Techno-Functional (Fechnical)
  3. 3.  White Papers ◦ http://www.slideshare.net/mvallamp Email: mvallamp@yahoo.com Blog: http://mvallamp.blogspot.com Linked in Group Leader: DBA Manager Oracle Alumni Admin for content: 5000 members
  4. 4.  Performance Issues Blame Blame The Business Blame the Functional Team Blame the Developer Blame the DBA Conclusions
  5. 5.  Performance Issues ◦ Described in vague terms  The System is slow  The disk is slow  The network is slow  The SQL sucks  The server sucks  The problem is in the Application server  The database is slow  The code is awful ◦ My favorite  Slow as molasses
  6. 6.  Issue Approach ◦ Sound sophisticated but not really effective (My opinion)  The BCHR is low or high  We need to run Stats-pack  AWR is better  No, ASH is better  “My” method is better  Have you run statistics?  Have you rebuilt the indexes?  Let us take a look at the top 10 SQL statements  The average physical I/O is high
  7. 7.  Let us Blame somebody Let us not leave anybody out Let us Blame ◦ The Business People ◦ The Functional Team ◦ The Developers ◦ The DBA’s
  8. 8.  Manufacturing Company In the Midwest Migrated from Mainframe to Oracle Applications (11.5.10) Did Big Bang implementation of Manufacturing, Order Management and Financials System was barely usable Flew there the day they went live after receiving panic calls
  9. 9.  Implementation vendor was there Another “expert” vendor was there ◦ BCHR and Block Size changes were being discussed Every theory possible was floating around Listened for an hour to the issues Asked a basic question ◦ Are you losing money and if so why?
  10. 10.  $$$$ ◦ We are losing money ◦ We can’t take orders ◦ Order Management is not able to take orders It is taking 10 minutes to enter a 6 line order It used to take less than a minute in the mainframe Good place to start They can pay my bills if only they make money
  11. 11.  Traced the Order Management process ◦ Entering a header ◦ Entering a line Line Trace ◦ Entering a line was taking 1 minute and a half Most of the time was spent on calculating the price of an item Enter “Advanced” Pricing
  12. 12.  Query was 9 pages long Query executed 319,000 Logical I/O’s
  13. 13.  Why are you using advanced pricing? We can make more money Let us try Simple Pricing Turned off Advanced Pricing 6 line Order with Simple Pricing took 1 minute Interesting!!!
  14. 14.  Advanced Pricing Parameters for determining price ◦ Checks Overall credit Limit (AR) ◦ Checks outstanding balances (AR) ◦ Checks inventory levels (Inv) ◦ Checks overall order totals (Current Order) ◦ Checks payment history (AR)
  15. 15.  Can we use Simple Pricing instead of Advanced Pricing? Enter the “Business” Discussion with COO ◦ Why are you using Advanced Pricing?  We can make more money. ◦ How much more money can you make with Advanced Pricing instead of Simple Pricing?  Don’t know
  16. 16.  Potential Upside on a 1billion USD in revenue ◦ 4 million a year if we are lucky Investment required to make the system perform at acceptable levels ◦ Cost – 10 to 15 million dollars  Standby Reporting  Archive/Purge  Hardware Investments  Software Investments  Investments in people, processes etc.
  17. 17. Probability 50% Discount Rate 5% Year1 Year2 Year3 Year4 Year5 Revenue Upside $4,000,000 $4,000,000 $4,000,000 $4,000,000 $4,000,000NPV $17,317,907 NPV for 5 YearsProbableRevenue $8,658,953 NPV times the ProbabilityInvestmentRequired $15,000,000 Capital Investment Required. Depreciation not included.Profit ($6,341,047) Revenue-Cost Incurred
  18. 18.  Question to the COO ◦ Would you spend 15 million USD in today’s dollars for a potential upside of 4 million dollars ◦ Economic forecast is gloomy ◦ Interest and Borrowing costs are high Right Answer, No.
  19. 19.  Changed from Advanced Pricing to Simple Pricing The system was fast in performing transactions System had over-capacity Even in month end, the system did not even flinch Performance team disbanded Rebuilding Indexes and tables into bigger block table-spaces got a yawn Solved in 2 days
  20. 20.  There is not enough hardware in the world to support an irrational business requirement Ask why complex functionality is required when simple functionality will do Ask the “Business” to justify need for “Advanced Functionality” Solution ◦ Switch to Simple Pricing ◦ System Stabilized ◦ Project started on Monday Morning ◦ Project Ended Wednesday Afternoon ◦ Back Home Wednesday night
  21. 21.  Manufacturing Company in Ohio Migrated to Oracle Applications Severe Performance Issues Order Management is slow Call center for customers is just horrific Each call taking 10-15 minutes when before each call during the mainframe was taking 2- 3 minutes Customer was using Configurator and BOM
  22. 22.  Method-R Traced the Order Management Process which uses the Configurator Identified the SQL Statement that was taking a long time
  23. 23.  SQL statement was performing close to 900,000 Logical I/O’s per execution SQL Statement was getting executed for every item in the BOM that makes up an order line The more complex the BOM, the more items in the BOM, the slower the process The SQL Statement was determining whether the item needed to be displayed in the invoice for the customer and in the packing slip
  24. 24.  1. SELECT x 2. FROM BOM_BILL_OF_MATERIALS BOM, BOM_INVENTORY_COMPONENTS COMP, 3. MTL_SYSTEM_ITEMS_B MTL, MTL_SYSTEM_ITEMS_B MTL_BOM, MTL_PARAMETERS MP 4. WHERE BOM.COMMON_BILL_SEQUENCE_ID = COMP.BILL_SEQUENCE_ID 5. AND COMP.COMPONENT_ITEM_ID = MTL.INVENTORY_ITEM_ID 6. AND MTL.ORGANIZATION_ID = MP.ORGANIZATION_ID 7. AND (MTL_BOM.SEGMENT1 LIKE %_ITEMS 8. OR MTL_BOM.SEGMENT1 LIKE %_ITEM 9. OR MTL_BOM.SEGMENT1 LIKE %_ITEMS2 10. OR MTL.BOM_ITEM_TYPE = :B3 11. OR MTL.SEGMENT1 LIKE OC_%) 12. AND BOM.ASSEMBLY_ITEM_ID = MTL_BOM.INVENTORY_ITEM_ID 13. AND BOM.ORGANIZATION_ID = MP.ORGANIZATION_ID 14. AND BOM.ORGANIZATION_ID = MTL_BOM.ORGANIZATION_ID 15. AND MP.ORGANIZATION_CODE = :B2 16. AND NVL(COMP.EFFECTIVITY_DATE, SYSDATE) <= SYSDATE 17. AND NVL(COMP.DISABLE_DATE, SYSDATE) >= SYSDATE 18. AND COMP.COMPONENT_ITEM_ID = :B1
  25. 25.  Should we tune the SQL? A couple of Predicates were using leading wildcards ◦ AND (MTL_BOM.SEGMENT1 LIKE %_ITEMS ◦ OR MTL_BOM.SEGMENT1 LIKE %_ITEM ◦ OR MTL_BOM.SEGMENT1 LIKE %_ITEMS2 ◦ OR MTL.BOM_ITEM_TYPE = :B3 ◦ OR MTL.SEGMENT1 LIKE OC_%)
  26. 26.  First instinct ◦ Blame the developer Met with the developer ◦ He said that was the functional spec Met with the functional person ◦ Asked them why they were looking for  %ITEM  %ITEMS  %ITEMS2  OC_%
  27. 27.  Functional Person explanation ◦ Oh, it is a convention we use ◦ If the item type ends in ITEM, ITEMS or ITEMS2 or starts with OC, then display it ◦ Otherwise, don’t display it ◦ They were proud of it too
  28. 28.  Should we create a reverse string functional index and switch the predicates? Umm..No
  29. 29.  Called a meeting ◦ Business Users ◦ Functional Team ◦ Developers ◦ DBA’s Took a dictionary to the meeting ◦ Asked them to find words that ended with M or S quickly ◦ Everybody was looking as if I was not sane I explained that was what they were asking the system
  30. 30.  Created a flexfield with a Y or N at the inventory item level Updated the field to indicate the status Rewrote the SQL Statement to use the flag instead of the convention SQL now used 40 Logical I/O’s System speed was so fast that the productivity of the call center was up.
  31. 31.  Try to answer the SQL statement yourself as if you did not have a system Would it make sense? If no, then the system is probably going to be bad at it too and will be slow Systems can process and automate irrational requests The cost and pain is going to be high System can do dumb but slow (Exadata anybody?)
  32. 32.  Manufacturing Company in Illinois Customer had implemented Oracle Applications Shipping Application is slow SR with Oracle Support going nowhere Cannot determine if it is custom or standard
  33. 33.  Method-R Traced the Shipping Process Each Line in the shipping screen was taking 20-25 seconds Identified the SQL Statement that was taking a long time Each execution was taking 650,000 Logical I/O’s.
  34. 34. SELECT fds.short_textFROM wsh_delivery_assignments wda,wsh_delivery_details wsh,fnd_attached_docs_form_vl fad,fnd_documents_short_text fdsWHERE wsh.released_status = YAND wsh.delivery_detail_id = wda.delivery_detail_idAND wda.delivery_id = 124901AND wsh.source_code = OEAND fad.datatype_name = Short TextAND fad.usage_type = OAND fad.function_name = OEXOEORDAND fad.entity_name = OE_ORDER_HEADERSAND fds.media_id = fad.media_idAND fad.pk1_value = wsh.source_header_idAND ROWNUM = 1
  35. 35.  1 - filter(ROWNUM=1) 2 - filter("WDA"."DELIVERY_ID"=124901) 13 - access("DET"."DATA_OBJECT_CODE"=OE_ORDER_HEADERS AND "DET"."LANGUAGE"=USERENV(LANG)) 15 - access("AF"."FUNCTION_NAME"=OEXOEORD) 16 - filter("DCU"."ENABLED_FLAG"=Y) 17 - access("DCU"."ATTACHMENT_FUNCTION_ID"="AF"."ATTACHMENT_FUNCTION_ID") 18 - filter("D"."USAGE_TYPE"=O AND "DCU"."CATEGORY_ID"="D"."CATEGORY_ID") 19 - access("D"."DATATYPE_ID"="DAT"."DATATYPE_ID" AND "DAT"."LANGUAGE"=USERENV(LANG) AND "DAT"."USER_NAME"=Short Text) 20 - access("DCT"."CATEGORY_ID"="D"."CATEGORY_ID" AND "DCT"."LANGUAGE"=USERENV(LANG)) 22 - access("DT"."DOCUMENT_ID"="D"."DOCUMENT_ID" AND "DT"."LANGUAGE"=USERENV(LANG)) 24 - access("FDS"."MEDIA_ID"="DT"."MEDIA_ID") 25 - filter("AD"."ENTITY_NAME"=OE_ORDER_HEADERS) 26 - access("D"."DOCUMENT_ID"="AD"."DOCUMENT_ID") 27 - filter("WSH"."RELEASED_STATUS"=Y AND "WSH"."SOURCE_CODE"=OE) 28 - access("WSH"."SOURCE_HEADER_ID"=TO_NUMBER("AD"."PK1_VALUE")) 29 - access("WSH"."DELIVERY_DETAIL_ID"="WDA"."DELIVERY_DETAIL_ID")
  36. 36.  Oracle Financials has flexfields which have multiple purposes They have to be varchar2 It is a common practice to store key values in these columns Key values tend to be numbers Joins are made to flexfields from these keyfield If you use an implicit type conversion happening in the plan, this is usually the issue
  37. 37. SELECT fds.short_textFROM wsh_delivery_assignments wda,wsh_delivery_details wsh,fnd_attached_docs_form_vl fad,fnd_documents_short_text fdsWHERE wsh.released_status = YAND wsh.delivery_detail_id = wda.delivery_detail_idAND wda.delivery_id = 1555559AND wsh.source_code = OEAND fad.datatype_name = Short TextAND fad.usage_type = OAND fad.function_name = OEXOEORDAND fad.entity_name = OE_ORDER_HEADERSAND fds.media_id = fad.media_idAND fad.pk1_value = to_char(wsh.source_header_id)AND ROWNUM = 1
  38. 38.  Query now executes 700 Logical I/O’s down from 650,000 LIO’s Query now has sub-second response Productivity is now way up. Life is good
  39. 39.  Known he was joining a varchar to a number field Recognize that it could be an issue Checked the number of Logical I/O’s it was doing Examined the explain plan Can we have a “lint” for SQL?
  40. 40.  Customer was a food distributor in Illinois They were going live on Oracle Financials They had several customizations which they were rolling out Customizations were released as a series of scripts The scripts were rolled out at 2:30 PM Testing would start
  41. 41.  Performance was horrible Find Customers, Find Orders and Find Shipment forms would take forever The next day the system performance was fine Everything was fast Everybody were stunned ◦ Client DBA team ◦ Oracle Consulting Team
  42. 42.  Traced the processes using 10046 A lot of joins to custom temporary tables were observed for queries with poor performance A little bit more digging and almost all performance issues were related to the temporary tables So what is that with these temporary tables that performed well the day after they were deployed
  43. 43.  After a few hours of searching, did a quick explain plan in SQL Developer Noticed the “Rule” in the explain plan Wait a minute – we should be using CBO. This is the 9i database Hmm..Let us look at the temporary tables Aaah..ah
  44. 44.  Stats on temporary tables were not being collected as a part of the code deployment A nightly process that gathered stats on all objects that did not have statistics gathered the rest That explained why performance was poor the day the code was deployed and better the day after
  45. 45.  This was interesting Whose job was it to have the stats gathering script as a part of the deployment Was it the DBA? Or the developer? It is a moot point when IT is on the line Is it the linebacker or the safety’s fault in a busted play? Who cares if you lose?
  46. 46.  Use Method-R to get to the root cause Use other methods to confirm the problem ◦ Statspack ◦ Grid Control ◦ Homegrown scripts Avoid your first instinct
  47. 47.  First Instinct ◦ Try to tune the SQL ◦ Try to solve the problem ◦ Wrong Approach Correct Approach ◦ Identify blame ◦ Assign Blame (Important) ◦ Ask the “makes sense” question ◦ Will it make us money? ◦ If not, don’t do it
  48. 48.  Eliminate the problem Train the user not to create the problem Reschedule the problem Tune/Optimize the problem Relocate the problem

×