Enough Blame for System Performance Issues

1,221 views

Published on

Published in: Technology
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
1,221
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
0
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Enough Blame for System Performance Issues

  1. 1. Mahesh Vallampati
  2. 2.  Mahesh Vallampati ◦ Career  Director of Oracle Applications and Technologies at Coach America, the Bus Company.  Sales and Consulting at Hotsos and SmartDog  Director of IT Database Services at Ceva Logistics  Sales and Consulting at Oracle for 9 years ◦ Education  Courses in Business at Houston Community College  Master’s in EE from Texas A&M University Career Focus ◦ Used to be DBA ◦ Now Techno-Functional (Fechnical)
  3. 3.  White Papers ◦ http://www.slideshare.net/mvallamp Email: mvallamp@yahoo.com Blog: http://mvallamp.blogspot.com Linked in Group Leader: DBA Manager Oracle Alumni Admin for content: 5000 members
  4. 4.  Performance Issues Blame Blame The Business Blame the Functional Team Blame the Developer Blame the DBA Conclusions
  5. 5.  Performance Issues ◦ Described in vague terms  The System is slow  The disk is slow  The network is slow  The SQL sucks  The server sucks  The problem is in the Application server  The database is slow  The code is awful ◦ My favorite  Slow as molasses
  6. 6.  Issue Approach ◦ Sound sophisticated but not really effective (My opinion)  The BCHR is low or high  We need to run Stats-pack  AWR is better  No, ASH is better  “My” method is better  Have you run statistics?  Have you rebuilt the indexes?  Let us take a look at the top 10 SQL statements  The average physical I/O is high
  7. 7.  Let us Blame somebody Let us not leave anybody out Let us Blame ◦ The Business People ◦ The Functional Team ◦ The Developers ◦ The DBA’s
  8. 8.  Manufacturing Company In the Midwest Migrated from Mainframe to Oracle Applications (11.5.10) Did Big Bang implementation of Manufacturing, Order Management and Financials System was barely usable Flew there the day they went live after receiving panic calls
  9. 9.  Implementation vendor was there Another “expert” vendor was there ◦ BCHR and Block Size changes were being discussed Every theory possible was floating around Listened for an hour to the issues Asked a basic question ◦ Are you losing money and if so why?
  10. 10.  $$$$ ◦ We are losing money ◦ We can’t take orders ◦ Order Management is not able to take orders It is taking 10 minutes to enter a 6 line order It used to take less than a minute in the mainframe Good place to start They can pay my bills if only they make money
  11. 11.  Traced the Order Management process ◦ Entering a header ◦ Entering a line Line Trace ◦ Entering a line was taking 1 minute and a half Most of the time was spent on calculating the price of an item Enter “Advanced” Pricing
  12. 12.  Query was 9 pages long Query executed 319,000 Logical I/O’s
  13. 13.  Why are you using advanced pricing? We can make more money Let us try Simple Pricing Turned off Advanced Pricing 6 line Order with Simple Pricing took 1 minute Interesting!!!
  14. 14.  Advanced Pricing Parameters for determining price ◦ Checks Overall credit Limit (AR) ◦ Checks outstanding balances (AR) ◦ Checks inventory levels (Inv) ◦ Checks overall order totals (Current Order) ◦ Checks payment history (AR)
  15. 15.  Can we use Simple Pricing instead of Advanced Pricing? Enter the “Business” Discussion with COO ◦ Why are you using Advanced Pricing?  We can make more money. ◦ How much more money can you make with Advanced Pricing instead of Simple Pricing?  Don’t know
  16. 16.  Potential Upside on a 1billion USD in revenue ◦ 4 million a year if we are lucky Investment required to make the system perform at acceptable levels ◦ Cost – 10 to 15 million dollars  Standby Reporting  Archive/Purge  Hardware Investments  Software Investments  Investments in people, processes etc.
  17. 17. Probability 50% Discount Rate 5% Year1 Year2 Year3 Year4 Year5 Revenue Upside $4,000,000 $4,000,000 $4,000,000 $4,000,000 $4,000,000NPV $17,317,907 NPV for 5 YearsProbableRevenue $8,658,953 NPV times the ProbabilityInvestmentRequired $15,000,000 Capital Investment Required. Depreciation not included.Profit ($6,341,047) Revenue-Cost Incurred
  18. 18.  Question to the COO ◦ Would you spend 15 million USD in today’s dollars for a potential upside of 4 million dollars ◦ Economic forecast is gloomy ◦ Interest and Borrowing costs are high Right Answer, No.
  19. 19.  Changed from Advanced Pricing to Simple Pricing The system was fast in performing transactions System had over-capacity Even in month end, the system did not even flinch Performance team disbanded Rebuilding Indexes and tables into bigger block table-spaces got a yawn Solved in 2 days
  20. 20.  There is not enough hardware in the world to support an irrational business requirement Ask why complex functionality is required when simple functionality will do Ask the “Business” to justify need for “Advanced Functionality” Solution ◦ Switch to Simple Pricing ◦ System Stabilized ◦ Project started on Monday Morning ◦ Project Ended Wednesday Afternoon ◦ Back Home Wednesday night
  21. 21.  Manufacturing Company in Ohio Migrated to Oracle Applications Severe Performance Issues Order Management is slow Call center for customers is just horrific Each call taking 10-15 minutes when before each call during the mainframe was taking 2- 3 minutes Customer was using Configurator and BOM
  22. 22.  Method-R Traced the Order Management Process which uses the Configurator Identified the SQL Statement that was taking a long time
  23. 23.  SQL statement was performing close to 900,000 Logical I/O’s per execution SQL Statement was getting executed for every item in the BOM that makes up an order line The more complex the BOM, the more items in the BOM, the slower the process The SQL Statement was determining whether the item needed to be displayed in the invoice for the customer and in the packing slip
  24. 24.  1. SELECT x 2. FROM BOM_BILL_OF_MATERIALS BOM, BOM_INVENTORY_COMPONENTS COMP, 3. MTL_SYSTEM_ITEMS_B MTL, MTL_SYSTEM_ITEMS_B MTL_BOM, MTL_PARAMETERS MP 4. WHERE BOM.COMMON_BILL_SEQUENCE_ID = COMP.BILL_SEQUENCE_ID 5. AND COMP.COMPONENT_ITEM_ID = MTL.INVENTORY_ITEM_ID 6. AND MTL.ORGANIZATION_ID = MP.ORGANIZATION_ID 7. AND (MTL_BOM.SEGMENT1 LIKE %_ITEMS 8. OR MTL_BOM.SEGMENT1 LIKE %_ITEM 9. OR MTL_BOM.SEGMENT1 LIKE %_ITEMS2 10. OR MTL.BOM_ITEM_TYPE = :B3 11. OR MTL.SEGMENT1 LIKE OC_%) 12. AND BOM.ASSEMBLY_ITEM_ID = MTL_BOM.INVENTORY_ITEM_ID 13. AND BOM.ORGANIZATION_ID = MP.ORGANIZATION_ID 14. AND BOM.ORGANIZATION_ID = MTL_BOM.ORGANIZATION_ID 15. AND MP.ORGANIZATION_CODE = :B2 16. AND NVL(COMP.EFFECTIVITY_DATE, SYSDATE) <= SYSDATE 17. AND NVL(COMP.DISABLE_DATE, SYSDATE) >= SYSDATE 18. AND COMP.COMPONENT_ITEM_ID = :B1
  25. 25.  Should we tune the SQL? A couple of Predicates were using leading wildcards ◦ AND (MTL_BOM.SEGMENT1 LIKE %_ITEMS ◦ OR MTL_BOM.SEGMENT1 LIKE %_ITEM ◦ OR MTL_BOM.SEGMENT1 LIKE %_ITEMS2 ◦ OR MTL.BOM_ITEM_TYPE = :B3 ◦ OR MTL.SEGMENT1 LIKE OC_%)
  26. 26.  First instinct ◦ Blame the developer Met with the developer ◦ He said that was the functional spec Met with the functional person ◦ Asked them why they were looking for  %ITEM  %ITEMS  %ITEMS2  OC_%
  27. 27.  Functional Person explanation ◦ Oh, it is a convention we use ◦ If the item type ends in ITEM, ITEMS or ITEMS2 or starts with OC, then display it ◦ Otherwise, don’t display it ◦ They were proud of it too
  28. 28.  Should we create a reverse string functional index and switch the predicates? Umm..No
  29. 29.  Called a meeting ◦ Business Users ◦ Functional Team ◦ Developers ◦ DBA’s Took a dictionary to the meeting ◦ Asked them to find words that ended with M or S quickly ◦ Everybody was looking as if I was not sane I explained that was what they were asking the system
  30. 30.  Created a flexfield with a Y or N at the inventory item level Updated the field to indicate the status Rewrote the SQL Statement to use the flag instead of the convention SQL now used 40 Logical I/O’s System speed was so fast that the productivity of the call center was up.
  31. 31.  Try to answer the SQL statement yourself as if you did not have a system Would it make sense? If no, then the system is probably going to be bad at it too and will be slow Systems can process and automate irrational requests The cost and pain is going to be high System can do dumb but slow (Exadata anybody?)
  32. 32.  Manufacturing Company in Illinois Customer had implemented Oracle Applications Shipping Application is slow SR with Oracle Support going nowhere Cannot determine if it is custom or standard
  33. 33.  Method-R Traced the Shipping Process Each Line in the shipping screen was taking 20-25 seconds Identified the SQL Statement that was taking a long time Each execution was taking 650,000 Logical I/O’s.
  34. 34. SELECT fds.short_textFROM wsh_delivery_assignments wda,wsh_delivery_details wsh,fnd_attached_docs_form_vl fad,fnd_documents_short_text fdsWHERE wsh.released_status = YAND wsh.delivery_detail_id = wda.delivery_detail_idAND wda.delivery_id = 124901AND wsh.source_code = OEAND fad.datatype_name = Short TextAND fad.usage_type = OAND fad.function_name = OEXOEORDAND fad.entity_name = OE_ORDER_HEADERSAND fds.media_id = fad.media_idAND fad.pk1_value = wsh.source_header_idAND ROWNUM = 1
  35. 35.  1 - filter(ROWNUM=1) 2 - filter("WDA"."DELIVERY_ID"=124901) 13 - access("DET"."DATA_OBJECT_CODE"=OE_ORDER_HEADERS AND "DET"."LANGUAGE"=USERENV(LANG)) 15 - access("AF"."FUNCTION_NAME"=OEXOEORD) 16 - filter("DCU"."ENABLED_FLAG"=Y) 17 - access("DCU"."ATTACHMENT_FUNCTION_ID"="AF"."ATTACHMENT_FUNCTION_ID") 18 - filter("D"."USAGE_TYPE"=O AND "DCU"."CATEGORY_ID"="D"."CATEGORY_ID") 19 - access("D"."DATATYPE_ID"="DAT"."DATATYPE_ID" AND "DAT"."LANGUAGE"=USERENV(LANG) AND "DAT"."USER_NAME"=Short Text) 20 - access("DCT"."CATEGORY_ID"="D"."CATEGORY_ID" AND "DCT"."LANGUAGE"=USERENV(LANG)) 22 - access("DT"."DOCUMENT_ID"="D"."DOCUMENT_ID" AND "DT"."LANGUAGE"=USERENV(LANG)) 24 - access("FDS"."MEDIA_ID"="DT"."MEDIA_ID") 25 - filter("AD"."ENTITY_NAME"=OE_ORDER_HEADERS) 26 - access("D"."DOCUMENT_ID"="AD"."DOCUMENT_ID") 27 - filter("WSH"."RELEASED_STATUS"=Y AND "WSH"."SOURCE_CODE"=OE) 28 - access("WSH"."SOURCE_HEADER_ID"=TO_NUMBER("AD"."PK1_VALUE")) 29 - access("WSH"."DELIVERY_DETAIL_ID"="WDA"."DELIVERY_DETAIL_ID")
  36. 36.  Oracle Financials has flexfields which have multiple purposes They have to be varchar2 It is a common practice to store key values in these columns Key values tend to be numbers Joins are made to flexfields from these keyfield If you use an implicit type conversion happening in the plan, this is usually the issue
  37. 37. SELECT fds.short_textFROM wsh_delivery_assignments wda,wsh_delivery_details wsh,fnd_attached_docs_form_vl fad,fnd_documents_short_text fdsWHERE wsh.released_status = YAND wsh.delivery_detail_id = wda.delivery_detail_idAND wda.delivery_id = 1555559AND wsh.source_code = OEAND fad.datatype_name = Short TextAND fad.usage_type = OAND fad.function_name = OEXOEORDAND fad.entity_name = OE_ORDER_HEADERSAND fds.media_id = fad.media_idAND fad.pk1_value = to_char(wsh.source_header_id)AND ROWNUM = 1
  38. 38.  Query now executes 700 Logical I/O’s down from 650,000 LIO’s Query now has sub-second response Productivity is now way up. Life is good
  39. 39.  Known he was joining a varchar to a number field Recognize that it could be an issue Checked the number of Logical I/O’s it was doing Examined the explain plan Can we have a “lint” for SQL?
  40. 40.  Customer was a food distributor in Illinois They were going live on Oracle Financials They had several customizations which they were rolling out Customizations were released as a series of scripts The scripts were rolled out at 2:30 PM Testing would start
  41. 41.  Performance was horrible Find Customers, Find Orders and Find Shipment forms would take forever The next day the system performance was fine Everything was fast Everybody were stunned ◦ Client DBA team ◦ Oracle Consulting Team
  42. 42.  Traced the processes using 10046 A lot of joins to custom temporary tables were observed for queries with poor performance A little bit more digging and almost all performance issues were related to the temporary tables So what is that with these temporary tables that performed well the day after they were deployed
  43. 43.  After a few hours of searching, did a quick explain plan in SQL Developer Noticed the “Rule” in the explain plan Wait a minute – we should be using CBO. This is the 9i database Hmm..Let us look at the temporary tables Aaah..ah
  44. 44.  Stats on temporary tables were not being collected as a part of the code deployment A nightly process that gathered stats on all objects that did not have statistics gathered the rest That explained why performance was poor the day the code was deployed and better the day after
  45. 45.  This was interesting Whose job was it to have the stats gathering script as a part of the deployment Was it the DBA? Or the developer? It is a moot point when IT is on the line Is it the linebacker or the safety’s fault in a busted play? Who cares if you lose?
  46. 46.  Use Method-R to get to the root cause Use other methods to confirm the problem ◦ Statspack ◦ Grid Control ◦ Homegrown scripts Avoid your first instinct
  47. 47.  First Instinct ◦ Try to tune the SQL ◦ Try to solve the problem ◦ Wrong Approach Correct Approach ◦ Identify blame ◦ Assign Blame (Important) ◦ Ask the “makes sense” question ◦ Will it make us money? ◦ If not, don’t do it
  48. 48.  Eliminate the problem Train the user not to create the problem Reschedule the problem Tune/Optimize the problem Relocate the problem

×