553: Oracle Database Performance: Are Database Users Telling Me The Truth?
•
1 like•944 views
This document discusses a method for accurately diagnosing database performance issues based on a case study. The method involves setting goals, measuring key metrics over time through snapshots, graphing the data, comparing across time periods, and analyzing to identify bottlenecks. It emphasizes getting the full context by questioning users and looking directly at database metrics, as users may perceive issues differently than what the data shows. Following this method can help identify root causes and times of problems rather than wasting time on incorrect assumptions. Tools like AWR, Statspack and real-time views can then be leveraged in diagnosis.
More Related Content
Similar to 553: Oracle Database Performance: Are Database Users Telling Me The Truth?
Similar to 553: Oracle Database Performance: Are Database Users Telling Me The Truth? (20)
More from Alfredo Krieg
More from Alfredo Krieg (11)
Recently uploaded
Recently uploaded (20)
553: Oracle Database Performance: Are Database Users Telling Me The Truth?
- 1. Oracle Database Performance: Are Database Users Telling Me The Truth? Session # 553 REMINDER Check in on the COLLABORATE mobile app Prepared by: Alfredo Krieg Database Administrator The Sherwin Williams Company
- 2. Agenda ■ Introduction ■ Performance Challenge ■ Case Study ■ The Method – Set Goals – Measure– Measure – Graph – Compare – Analyze ■ Anticipate ■ Tools ■ Summary
- 3. Who am I? ■ Alfredo Krieg (Alfredo.Krieg.Villa@sherwin.com) ■ Senior Oracle Cloud Administrator at The Sherwin Williams Company based in Cleveland, Oh ■ OEM Cloud Control 12c and Database Performance Tuning■ OEM Cloud Control 12c and Database Performance Tuning ■ Oracle Technologies since 2004 & 11g Certified ■ Blog bitkode.blogspot.com
- 4. Agenda ■ Introduction ■ Performance Challenge ■ Case Study ■ The Method – Set Goals – Measure– Measure – Graph – Compare – Analyze ■ Anticipate ■ Tools ■ Summary
- 5. Performance Challenge ■ Difficult without a performance tuning method ■ Multiple metrics, ratios and performance data to look into ■ DBA’s sometimes rely in what users experience and communicate ■ Analyze wrong snapshots, not when problem really happened ■ Are DB users telling me the truth? What to do?
- 6. Agenda ■ Introduction ■ Performance Challenge ■ Case Study ■ The Method – Set Goals – Measure– Measure – Graph – Compare – Analyze ■ Anticipate ■ Tools ■ Summary
- 7. Case Study ■ Friday 10 am in the morning, DBA receives a call from a database user stating that the production DB is slow. DBA questions the user in order to better understand the situation: ▪ When did the performance issue start? ▪ Has this happened before? ▪ Is it just your session or are other users affected as well? ■ Database user responds using his/her best knowledge of the■ Database user responds using his/her best knowledge of the situation ▪ When did it start? I arrived today at 9 am and it was already slow! ▪ Has this happened before? Yes, it happened last Friday and the Friday before that too! ▪ Is it just your session or are other users affected as well? Seems like the entire department is being affected!
- 8. Case Study ■ DBA trusts the user and starts comparing previous AWR/Statspack snapshots against the one contained between 9:30 am and 10 am. DBA gathers a couple of reports and looks at the ratios, CPU utilization and some other performance metrics finding not much change in the values when suddenly the user states that performance came back to normal and the call ends. “DBA Magic” ■ What happened?: ▪ There was no method ▪ Bottleneck was not identified ▪ Root cause was not identified ▪ No permanent fix applied
- 9. Case Study ■ Key data communicated by the user ▪ At 9 am the problem was present ▪ Last 2 Fridays as well ▪ Entire department impacted ■ Do you trust in user?, We need to look into the DB system to see who’s right!
- 10. Agenda ■ Introduction ■ Performance Challenge ■ Case Study ■ The Method – Set Goals – Measure– Measure – Graph – Compare – Analyze ■ Anticipate ■ Tools ■ Summary
- 11. The Method ■ Must establish the stop tuning criteria ■ Define the scope of tuning ■ CTD (Compulsive Tuning Disorder) Set Goals ■ CTD (Compulsive Tuning Disorder) “Find out if the DB is slow/How slow? (Find the Response Time)” “Identify the period of time and the bottleneck of the performance issue reported by the DB user” “Identify if this issue has any relationship with the issue of the last 2 Fridays”
- 12. Agenda ■ Introduction ■ Performance Challenge ■ Case Study ■ The Method – Set Goals – Measure– Measure – Graph – Compare – Analyze ■ Anticipate ■ Tools ■ Summary
- 13. The Method ■ Oracle’s Kernel Instrumentation (Wait Interface) ■ System views v$sysstat, v$system_event & v$sys_time_model (session level views can be used to troubleshoot performance Measure problems at session level) TIMED_STATISTICS=TRUE (modifiable) ■ Calculate DB Response Time for a period of time ▪ Service Time ms (Total DB CPU consumption) ▪ Queue Time ms (Total Non Idle wait time) ▪ User calls (Unit of Work) ▪ Snapshot’s time frame (i.e. 15 min, 30 min, 60 min, etc)
- 14. The Method ■ Queuing Theory (R = S + W) RT ms/uc = Service Time + Queue Time User Calls Measure ■ As per Oracle Documentation ▪ Response time refers to the amount of time Enterprise Server takes to return the results of a request to the user. The response time is affected by factors such as network bandwidth, number of users, number and type of requests submitted, and average think time.
- 15. The Method ■ Choose a Unit of Work ▪ User Calls (overall activity) ▪ Physical Reads/Writes (I/O bound) ▪ Logical Reads/Writes (CPU bound) Measure ■ User Calls as Per Oracle Documentation ▪ “User Calls” represents the number of logins, parses, or execute calls during the sample period and is an overall activity level monitor. http://bitkode.blogspot.com/2013/08/testing-user-calls.html http://shallahamer-orapub.blogspot.com/2010/05/understanding-user-calls.html
- 16. The Method ■ Take two snapshots from v$sysstat, v$system_event & v$sys_time_model and extract “Delta” values (Data Capture) ▪ Make use of AWR or Statspack (AWR requires license) verify retention! ▪ Extract AWR SQL script can be obtained from http://bitkode.blogspot.com/2014/02/extract-awr-data-to-build-response-time.html Measure http://bitkode.blogspot.com/2014/02/extract-awr-data-to-build-response-time.html ■ The smaller the time between snapshots the more accurate the measure (but higher the overhead?) ■ Calculate the Response Time of every delta and import the data into an spreadsheet
- 17. The Method ■ Calculate the Mode/Average Response Time of your system ■ The Mode is the value that the most often appears in a set of data, ■ Average can make you lose the clear picture of the DB response time (skew data). Median is good as well! Measure time (skew data). Median is good as well! ■ See the difference between average, mode & median!
- 18. Agenda ■ Introduction ■ Performance Challenge ■ Case Study ■ The Method – Set Goals – Measure– Measure – Graph – Compare – Analyze ■ Anticipate ■ Tools ■ Summary
- 19. The Method ■ Plot Column SNAP_TIME Against NON_IDLE_WAIT, DB_CPU, UC & RT_MS_PER_UC Graph ■ Great Visuals of Database Performance But still unclear!
- 20. Agenda ■ Introduction ■ Performance Challenge ■ Case Study ■ The Method – Set Goals – Measure– Measure – Graph – Compare – Analyze ■ Anticipate ■ Tools ■ Summary
- 21. The Method ■ Slice and Dice data to compare only last 3 Fridays Compare ■ Remember that RT’s Mode was 2220 ms/uc
- 22. The Method ■ Identify the time of the performance problem Compare ■ Spikes in Response Time & Non Idle Wait Time ■ Is slow?/How slow? 2.2 sec | > 40 sec
- 23. Agenda ■ Introduction ■ Performance Challenge ■ Case Study ■ The Method – Set Goals – Measure– Measure – Graph – Compare – Analyze ■ Anticipate ■ Tools ■ Summary
- 24. The Method ■ Focus on those 3 periods of time ▪ 5:30 am to 9:00 am ▪ 5:30 am to 9:00 am ▪ 7:00 am to 10:30 am ■ What changed when the performance “got bad”? Analyze ■ Use AWR/Statspack reports to Find Bottleneck
- 25. The Method ■ Find Configuration Problems, Bad SQL, OS problems Analyze ■ Sessions asking for the same buffers through multiple instances
- 26. The Method ■ What changed? Analyze
- 27. The Method Who was correct? ■ Both were right! But get always the whole picture of the issue. ■ Question your system!
- 28. The Method ■ Change application design ■ Increase network interconnect speed ■ Setup DB service Possible Solutions ■ Setup DB service ■ All solutions tied to company’s needs!
- 29. Agenda ■ Introduction ■ Performance Challenge ■ Case Study ■ The Method – Set Goals – Measure– Measure – Graph – Compare – Analyze ■ Anticipate ■ Tools ■ Summary
- 30. Anticipate ■ AWR & Statspack Not Real Time Data! ■ System Views v$sysstat, v$system_event & v$sys_time_model Close to Real Time ▪ Measure Response Time (Shrink time between Snapshots) ▪ Compare Against RT’s Mode/Median/Average ▪ Alert if Threshold Reached (Threshold not easy to define!)
- 31. Agenda ■ Introduction ■ Performance Challenge ■ Case Study ■ The Method – Set Goals – Measure– Measure – Graph – Compare – Analyze ■ Anticipate ■ Tools ■ Summary
- 32. Tools ■ Oracle Enterprise Manager ▪ Out of the box Metrics (RT per TXN) ▪ Adaptive Thresholds (fixed value, percent of maximum and significance level) ▪ Metric Extensions ■ 3rd Party Performance Tuning SW (cost)
- 33. Agenda ■ Introduction ■ Performance Challenge ■ Case Study ■ The Method – Set Goals – Measure– Measure – Graph – Compare – Analyze ■ Anticipate ■ Tools ■ Summary
- 34. Summary ■ Follow a method ■ Ask your DB system not just to users, know your system!, but also ask your users ■ Users experience different Response Times than DB system ■ Don’t waste your time by analyzing useless snapshots ■ Time is money! ■ If available, make use of provided tools
- 35. Want more? ■ Shallahamer, Craig ▪ Book: “Oracle Performance Firefighting” (ISBN 978-0-9841023-0-3) ▪ http://shallahamer-orapub.blogspot.mx/2010/11/average-challenge- part-1.html ■ Sztrik, Dr. János ▪ Basic Queuing Theory http://irh.inf.unideb.hu/~jsztrik/education/16/SOR_Main_Angol.pdf ■ Millsap, Cary ▪ Thinking clearly about performance ▪ http://method-r.com/papers/file/44-thinking-clearly-about-performance http://method-r.com/papers/doc_download/44-thinking-clearly-about- performance-cary-millsap
- 36. Thank You! Special thanks to: My manager & supervisor in The Sherwin Williams Company Friends and colleagues that helped with suggestions Friend and mentor from OR for all the support and suggestions My family to understand the extra time working
- 37. Please complete the session evaluation on the mobile app We appreciate your feedback and insight This box will have simplified instructions about how to complete the session evaluation online