Ash and awr deep dive hotsos


Published on

#HotSym14 Presentation on ASH and AWR

Published in: Technology

Ash and awr deep dive hotsos

  1. 1. How to Dive in Fast and Find the Right Answer to the Big Questions Kellyn Pot’Vin, Sr. Technical Consultant
  2. 2. A Little About Me:  ACE Director, Oak Table Member  Board of Directors for RMOUG  Conference Director for RMOUG  DB Track Lead for KSCOPE  Author and presenter at Oracle Open World, HotSos, Collaborate, KSCOPE and others…  Advocate for Women in Technology  Lives in Westminster, CO!
  3. 3. Bit of a Change What do I get questions on?  Work Smart, Don’t Work Hard…   ASH/AWR Reports in EM12c  Compare ADDM  SQL Monitor  ASH/AWR Reporting from the Command Line  Search SQL, (Least Used Feature of EM)  ASH Queries
  4. 4. The Road Trip Where do you want to go?  How will you get there? 
  5. 5. Cool….
  6. 6. Where You End up…
  7. 7. Stay on the Right Road Optimization- Tune for Time or You’re Wasting Time.  Know Your Goal(s)  Set a Stopping Point, avoid OTD, (Obsessive Tuning Disorder)  Do NOT Assume. Always do the Research and Have Data. 
  8. 8. Brief History       ASH= Active Session History AWR= Automatic Workload Repository Introduced in Oracle 10g Evolution to statspack, requests for performance reporting improvements. “Always on” approach to performance metrics with requirement of non-locking collection process. Requires Management Diagnostic Pack License from Oracle.
  9. 9. The Location in EM12c For Some of Today’s Presentation…
  10. 10. Running ASH Report from EM • • • • ASH is by time, not snapshot. Set start date and time. End date and time Generate report
  11. 11. HTML Format ASH
  12. 12. Main ASH Info
  13. 13. Top SQL, Top Sessions
  14. 14. Top SQL Details
  15. 15. Top Parallel, Top DB Files
  16. 16. How Often Are We Asked, “What Changed?” It ran fine last week, now it doesn’t!  ETL loads have changed, but no one has released any new code!  The DBA says there hasn’t been any parameter changes to this database, but I’m sure there have been.   Compare ADDM Resolution
  17. 17. What is Compare ADDM? Uses Two AWR snapshots  Creates Report that  Compares SQL that is common and different.  SGA changes  IO/CPU bound issues  Parameter Changes
  18. 18. What is in an ADDM Compare?  Report Includes Following:  Clear Demonstration of impact of change.  Recommendations to address issue.  Identifies causes behind change, (with limitations.)  Lists Regressed SQL, too!  Tip: If Installing to database for first time, (simple installation, nothing to concern about, just pkg to fulfill views) you must have preferred credentials SET of install will fail!
  19. 19. Executing a Report from EM12c Choose focus period  Choose to compare to an Offset, Baseline or Custom 
  20. 20. 2-3PM, 12th Vs. 13th Familiar interface with visual wait event comparisons.  Average # of sessions during each period are displayed. 
  21. 21. Detail Report  High level data, highlight for analysis and recommendations.
  22. 22. Regression SQL  Any performance degradation is noted with the down arrow icon:
  23. 23. Top Segments Causing IO Waits High Level data  Click on “Show Hot Object Breakdown” to see more detail. 
  24. 24. Resource Comparisons Comparisons of Memory, CPU, IO and Interconnect.  Memory Is there Virtual paging?  Memory Base Period  Memory Comparison Period 
  25. 25. Dashboard Provided for CPU, Memory, IO and Interconnect  Is something OTHER than Oracle the cause?
  26. 26. IO Bound Dashboard Base vs. comparison period  Temp reads/writes specified  Single block read latency 
  27. 27. SQL Monitor, EM12c Style One More way to identify performance issues.  Monitoring view ease for those less familiar with database performance. 
  28. 28. SQL Monitor Dashboard Status of Statement SQL_ID Wait Events Degree of Parallelism SQL Text
  29. 29. Exadata and Offloading  Drill down to specific statement within SQL Monitor will display offload efficiency per statement.
  30. 30. Full Detail of SQL Execution
  31. 31. View Report
  32. 32. SQL Monitor Report CLI SET LONG 1000000 SET LONGCHUNKSIZE 1000000 SET LINESIZE 1000 SET PAGESIZE 0 SET TRIM ON SET TRIMSPOOL ON SET ECHO OFF SET FEEDBACK OFF SELECT DBMS_SQLTUNE.report_sql_monitor( sql_id => '5vh6y3b7tnv8r', type => 'TEXT', report_level => 'ALL') AS report FROM dual;
  33. 33. Text Output of SQL Monitor
  34. 34. One of the Best & Least Used Features in EM Search SQL Problem Query
  35. 35. We Have the SQL_ID, What Next? 4v2tsp8dz0nhn is our SQL_ID Go to the EM Console, (Example is EM12c)
  36. 36. Search SQL Interface  Choose AWR Snapshots, (change Time Period), AWR Baselines and put SQL_ID
  37. 37. Click on Search SQL_ID link for SQL Details  Split up by tabs for Cursor, AWR, Baselines and SQL Tuning Sets  Plan Hash Value  Elapsed Time 
  38. 38. AWR Data
  39. 39. Snapshot IDs  Click on Snapshot ID and gather valuable data on resource usage during snapshot time or choose to view report.
  40. 40. AWR Report or Run ADDM Report
  41. 41. AWR and ASH from the CLI All DBAs should know how to do this!
  42. 42. Running Reports, Command Line $ORACLE_HOME/rdbms/admin/awrrpt.sql; $ORACLE_HOME/rdbms/admin/ashrpt.sql; $ORACLE_HOME/rdbms/admin/awrsqlrpt.sql; Less Known AWR Reports: awrinfo.sql General AWR Info awrddrpt.sql Comparison report between snapshots awrblmig.sql Migrates pre-11g baseline data into 11g Baseline tables. awrgrpt.sql RAC Aware AWR Report.
  43. 43. AWR Info Report Snapshot Interval Information  Basic Info on Instances and Nodes  No User or Application Schema info.  Space Usage by SYSAUX  WRH$ and Non- AWR Objects, ordered by size  Snapshot info and if any errors.  Advisor Tasks 
  44. 44. AWR Info Report
  45. 45. ASH Info Report @$ORACLE_HOME/rdbms/admin/awrrpt.sql; -Report Format: Text or HTML -Days to view snapshot IDs -Beginning and Ending Snapshot ID’s - Name of Report
  46. 46. ASH Report @$ORACLE_HOME/rdbms/admin/ashrpt.sql; -Report Format: Text or HTML. -Timestamp to being report from. -Duration in minutes. -Name of report.
  47. 47. SQL_ID Specific AWR Report
  48. 48. “Interesting Part”
  49. 49. Finale! Select * from table(dbms_xplan.display_awr(‘43mp3mjufgnkg’));
  50. 50. Querying ASH Data Directly More defined reporting  No need to pull full report  Detail on waits that are of interest  Join to non-AWR objects   Examples and Ideas…
  51. 51. V$ACTIVE_SESSION_HISTORY            SAMPLE_ID- This is a unique identifier within an ASH sample. SAMPLE_TIME- A unit of time used by Active Session History, (not to be confused with DB_TIME) USER_ID- Identifier for a user that’s executing the session. SESSION_ID- Same as the SID or Session ID and can be used to join to SID in other views/tables. SESSION_STATE- What was the state of the session when ASH recorded the sample. ON CPU/WAITING- The two session states in Active Session History. ON CPU is active, vs. Waiting, which is self-explanatory. EVENT- Type of event that the session is currently active or waiting on. TIME_WAITED- How long the session has been waiting if waiting. WAIT_TIME- Confusing- but this is populated by any wait time if the session is currently active and for the previous waits. SQL_ID- The unique identifier for the SQL statement being executed. SQL_CHILD_NUMBER-The cursor child number.
  52. 52. Session Averages select ROUND(RATIO_TO_REPORT(SUM(1)) OVER () * 100 ,2) PERCENTAGE, ash.session_type SESS_TYPE, session_state STATUS, decode(nvl(sql_id,'-1'),'-1','nonsql','sql') SQL_TYPE, count(distinct to_char(session_id)|| to_char(session_serial#)) SESS_CNT from v$active_session_history ash where sample_time > sysdate - 30/(24*60) and ( ( ash.session_state = 'ON CPU' ) or ( ash.session_type != 'BACKGROUND' ) ) group by ash.session_type, ash.session_state, decode(nvl(sql_id,'-1'),'-1','nonsql','sql') order by count(*) /
  53. 53. Session Avg. Output  Note the % of Background processes
  54. 54. Inspecting What col type for a20 select * from (select ash.SQL_ID , ash.SQL_PLAN_HASH_VALUE Plan_hash, type, sum(decode(ash.session_state,'ON CPU',1,0)) "CPU", sum(decode(ash.session_state,'WAITING',1,0)) "WAITING", sum(decode(ash.session_state,'WAITING', decode(wait_class, 'User I/O',1,0),0)) "IO WAIT" , sum(decode(ash.session_state,'WAITING', decode(wait_class, 'User I/O',1,0),0)) "IO" , sum(decode(ash.session_state,'WAITING', decode(wait_class, 'Concurrency',1,0))) "CONCURRENCY" , sum(decode(ash.session_state,'WAITING', decode(wait_class, 'Application',1,0))) "Application" , sum(decode(ash.session_state,'ON CPU',1,1)) "TOTAL" from v$active_session_history ash, audit_actions aud where SQL_ID is not NULL and ash.sql_opcode=aud.action and ash.sample_time > sysdate - &minutes /( 60*24) group by sql_id, SQL_PLAN_HASH_VALUE , order by sum(decode(session_state,'ON CPU',1,1)) desc ) where rownum < 5;
  55. 55. 10 Min. View of Waits by SQL_ID Choose Time in Minutes To Review, (10 in our example)  SQL_ID and Plan Hash Value Shown  Waits for CPU, Wait, IO Wait and others. 
  56. 56. Quantity of Events Occurred Over Small Amounts of Time Col event for a50 select event, count(1) from v$active_session_history where sample_time between to_date('21-FEB-14 01.43.00 PM','dd-MON-yy hh:mi:ss PM') and to_date('21-FEB-15 01.53.00 PM','dd-MON-yy hh:mi:ss PM') group by event order by event;
  57. 57. Results, Where to Focus?
  58. 58. Transaction Wait Detail col event for a22 col block_type for a18 col objn for a18 col otype for a10 col fn for 99 col sid for 9999 col bsid for 9999 col lm for 99 col p3 for 99999 col blockn for 99999 select to_char(sample_time,'HH:MI') st, substr(event,0,20) event, ash.session_id sid, mod(ash.p1,16) lm, ash.p2, ash.p3, nvl(o.object_name,ash.current_obj#) objn, substr(o.object_type,0,10) otype, CURRENT_FILE# fn, CURRENT_BLOCK# blockn, ash.SQL_ID, BLOCKING_SESSION bsid from v$active_session_history ash, all_objects o where event like 'enq: TX%' and o.object_id (+)= ash.CURRENT_OBJ# and sample_time > sysdate - 10/(60*24) Order by sample_time
  59. 59. Transaction Lock Output  What TX row locks are occurring!
  60. 60. Knowing What’s in the ASH Buffer • Deters from making assumptions on what data is being queried. • Know your samples!
  61. 61. Wait Events Across Nodes
  62. 62. Query top 10 SQL_ID’s in the last 10 minutes?
  63. 63. SQL_ID and CPU Usage
  64. 64. IO Waits by Object from ASH
  65. 65. SQL Text with ASH • SQL for most recent five minutes of sample data from ASH
  66. 66. SQL Results • SQL_ID, SQL Text, Sample Time that Process was captured in.
  67. 67. Average Activity- Graphed
  68. 68. Formatting and Setup accept hours prompt "hours (default 12) : " default 12 column f_hours new_value v_hours select &hours f_hours from dual; column f_secs new_value v_secs column f_samples new_value samples select 3600 f_secs from dual; select &v_secs f_samples from dual; --select &seconds f_secs from dual; column f_bars new_value v_bars select 5 f_bars from dual; column aas format 999.99 column f_graph new_value v_graph select 30 f_graph from dual; column graph format a30 column total format 99999 column npts format 99999 col waits for 99999 col cpu for 9999
  69. 69. select to_char(to_date(tday||' '||tmod*&v_secs,'YYMMDD SSSSS'),'DD-MON HH24:MI:SS') tm, samples npts,total/&samples aas, substr(substr(substr(rpad('+',round((cpu*&v_bars)/&samples),'+') || rpad('-',round((waits*&v_bars)/&samples),'-') || rpad(' ',p.value * &v_bars,' '),0,(p.value * &v_bars)) || p.value || substr(rpad('+',round((cpu*&v_bars)/&samples),'+') || rpad('-',round((waits*&v_bars)/&samples),'-') || rpad(' ',p.value * &v_bars,' '),(p.value * &v_bars),10) ,0,30) ,0,&v_graph)graph,total,cpu, waits from ( select to_char(sample_time,'YYMMDD')tday , trunc(to_char(sample_time,'SSSSS')/&v_secs) tmod , sum(decode(session_state,'ON CPU',1,decode(session_type,'BACKGROUND',0,1))) total , (max(sample_id) - min(sample_id) + 1 ) samples , sum(decode(session_state,'ON CPU' ,1,0)) cpu , sum(decode(session_type,'BACKGROUND',0,decode(session_state,'WAITING',1,0))) waits /* for waits I want to subtract out the BACKGROUND but for CPU I want to count everyon */ from v$active_session_history where sample_time > sysdate - &v_hours/24 group by trunc(to_char(sample_time,'SSSSS')/&v_secs), to_char(sample_time,'YYMMDD') union all select to_char(sample_time,'YYMMDD')tday , trunc(to_char(sample_time,'SSSSS')/&v_secs) tmod , sum(decode(session_state,'ON CPU',10,decode(session_type,'BACKGROUND',0,10))) total , (max(sample_id) - min(sample_id) + 1 ) samples , sum(decode(session_state,'ON CPU' ,10,0)) cpu , sum(decode(session_type,'BACKGROUND',0,decode(session_state,'WAITING',10,0))) waits /* for waits I want to subtract out the BACKGROUND but for CPU I want to count everyon */ from dba_hist_active_sess_history where sample_time > sysdate - &v_hours/24 and sample_time < (select min(sample_time) from v$active_session_history) group by trunc(to_char(sample_time,'SSSSS')/&v_secs), to_char(sample_time,'YYMMDD')) ash, v$parameter p where'cpu_count' order by to_date(tday||' '||tmod*&v_secs,'YYMMDD SSSSS') /
  70. 70. Pivot the Wait Events
  72. 72. Process Information SELECT * FROM ( SELECT /*+ PARALLEL */ count(*) AS count, user_id, program, module, sql_id FROM SYS.DBA_HIST_ACTIVE_SESS_HISTORY WHERE sample_time > TO_DATE('19-FEB-2014 03.00.00 PM','dd-MON-yy hh:mi:ss PM') AND sample_time < TO_DATE('19-FEB-2014 08.00.00 PM','ddMON-yy hh:mi:ss PM') AND program LIKE 'oracle@%' GROUP BY user_id, program, module, machine, sql_id ORDER BY count(*) desc ) WHERE rownum <= 20 /
  73. 73. Results of Process History
  74. 74. Tyler Muth ASH Mining Query
  75. 75. ASH Mining Output Additional Options: • Physical Read Averages • Physical Writes, (Max/Averages) • Redo Info • Login Info • Hard Parsing, etc.
  76. 76. Best Practice When Querying ASH Data       Keep it Simple and don’t reinvent the wheel. Samples are an alias for time, not for counts. Understand what is valuable and compare to packaged reports. Be aware on RAC of node specific data. Take care when querying Obj#, File# and Block#, (still issues in different versions…) Check the time that is available in buffer, don’t assume!
  77. 77. AWR/ASH Links/Blogs      Tyler Muth: Kyle Hailey, John Beresniewicz, Graham Wood: Mine- “For the Love of ASH and AWR” Karl Arao- Guy Harrison amples/
  78. 78. Connect with me-