Oracle AWR Data mining

9,029 views

Published on

The presentation on how to mine Oracle Automatic Workload Repository.

Published in: Technology, Education
1 Comment
17 Likes
Statistics
Notes
  • Hi Yury,

    can we get audio of this presentation?

    Thanks,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
9,029
On SlideShare
0
From Embeds
0
Number of Embeds
129
Actions
Shares
0
Downloads
745
Comments
1
Likes
17
Embeds 0
No embeds

No notes for slide
  • Trace Analyzer, also known as TRCANLZR or TRCA, is a tool provided by Oracle Server Technologies Center of Expertise - ST CoE. TRCA inputs one or several SQL trace(s) generated by Event 10046 and outputs a diagnostics report in two formats (html and text). These reports are commonly used to diagnose processes performing poorly.
  • REM File: yury1.sqlset echo off feed off lines 110 pages 9990REM Copyright (c) 2011, TPG, all rights reservedclear breakclear colclear computettitle offbtitle offundefine starthh24undefine endhh24undefinedaysagocol beginttim heading 'Begin|Interval' format a15 trunccol numtimes format 9,990 head 'Num|Times'col sql_id heading 'SQL ID' format a13col sumcputimeDelta heading 'SUM|CPU|Time|Delta' format 9,999,999,990col buffer_gets_Delta heading 'Buff|Gets' format 999,999,990col sumdiskreadsDelta heading 'SUM|Disk|Reads' format 9,999,990col sumiowaitDelta heading 'SUM|IO|Wait' format 9,999,999,990break on beginttim skip 1ttitle skip 1 center 'Top 10 SQL Delta Statistics &&daysago Days ago Hours &&starthh24 to &&endhh24' skip 2spool yury1.lisselect * from( selects.SQL_ID , sum( s.CPU_TIME_DELTA ) sumCPUTIMEDELTA , sum( s.DISK_READS_DELTA ) sumDISKREADSDELTA , sum( s.IOWAIT_DELTA ) sumiowaitDELTA , count( * ) numtimes from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p, DBA_HIST_SQLTEXT t where ( s.SNAP_ID = p.SNAP_ID ) and ( s.SQL_ID = t.SQL_ID ) and ( EXTRACT( HOUR FROM p.END_INTERVAL_TIME ) between &starthh24 and &endhh24 ) and ( t.COMMAND_TYPE != 47 ) -- Exclude PL/SQL blocks from output and ( p.END_INTERVAL_TIME between ( SYSDATE - &daysago ) and SYSDATE ) group bys.SQL_ID order by sum( s.CPU_TIME_DELTA ) desc )whererownum < 11/PROMPTspool offclear breakclear colclear computettitle offbtitle offset echo on feedback on lines 80 pages 60 term on
  • CREATE OR REPLACE function phy_get_secs (t1 in timestamp, t2 in timestamp)return number is sec_diff number;t_delta interval day to second;begint_delta := t2 - t1;sec_diff := ( extract(hour from t_delta)*60*60 + extract(minute from t_delta)*60 + extract(second from t_delta));return sec_diff;end;/show err
  • set pages 1000prompt SNAP_TIME_P|SNAP_TIME|SEC_D|VALUE_P|VALUE|VALUE_D|VALUE_PER_SECselect /*+ ORDERED FULL(st) */ LAG (st.SNAP_TIME) OVER (ORDER BY st.SNAP_TIME)||'|'||st.SNAP_TIME||'|'||trunc((st.SNAP_TIME - LAG (st.SNAP_TIME) OVER (ORDER BY st.SNAP_TIME))*24*60*60)||'|'|| LAG (ss.value) OVER (ORDER BY st.SNAP_TIME)||'|'||ss.value||'|'|| ( ss.value - LAG (ss.value) OVER (ORDER BY st.SNAP_TIME) )||'|'||trunc( ( ss.value - LAG (ss.value) OVER (ORDER BY st.SNAP_TIME) ) / ((st.SNAP_TIME - LAG (st.SNAP_TIME) OVER (ORDER BY st.SNAP_TIME))*24*60*60) ) fromv$databasevd,stats$snapshotst, V$STATNAME sn, STATS$SYSSTAT ss where 1=1 and ss.snap_id=st.snap_id and ss.instance_number=1 and ss.dbid=vd.dbid and sn.STATISTIC#=ss.STATISTIC# and sn.name='parse count (hard)' and st.SNAP_TIME between trunc(sysdate)-14 and trunc(sysdate) order by st.SNAP_TIME;select s.BEGIN_INTERVAL_TIME,s.END_INTERVAL_TIME, t.VALUE EVALUE, LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) BVALUE,t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) DVALUE,s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME DTIME, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) H, EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) M, EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) S, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) Secs, (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/(EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)) VAL_SECfrom DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT twhere 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = 'parse count (hard)'order bys.BEGIN_INTERVAL_TIME;CREATE OR REPLACE function phy_get_secs (t1 in timestamp, t2 in timestamp)return number is sec_diff number;t_delta interval day to second;begint_delta := t2 - t1;sec_diff := ( extract(hour from t_delta)*60*60 + extract(minute from t_delta)*60 + extract(second from t_delta));return sec_diff;end;/show errcolumn snap_date for a20select cast (s.BEGIN_INTERVAL_TIME as date) snap_date, (t.TIME_WAITED_MICRO-LAG (t.TIME_WAITED_MICRO) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/(t.TOTAL_WAITS-LAG (t.TOTAL_WAITS) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/1000 ms_per_ev, (t.TOTAL_WAITS-LAG (t.TOTAL_WAITS) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/(EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)) ev_SECfrom DBA_HIST_SNAPSHOT s, DBA_HIST_SYSTEM_EVENT twhere 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.EVENT_NAME = 'db file scattered read'order bys.BEGIN_INTERVAL_TIME;select sh.BEGIN_INTERVAL_TIME d ,trunc(st.TOTAL_WAITS-(LAG(st.TOTAL_WAITS) OVER (ORDER BY sh.BEGIN_INTERVAL_TIME))) tw_d ,trunc(st.TOTAL_TIMEOUTS-(LAG(st.TOTAL_TIMEOUTS) OVER (ORDER BY sh.BEGIN_INTERVAL_TIME))) tt_d ,trunc((st.TIME_WAITED_MICRO-(LAG(st.TIME_WAITED_MICRO) OVER (ORDER BY sh.BEGIN_INTERVAL_TIME)))/1000000) w_d_secsfrom DBA_HIST_SYSTEM_EVENT st, DBA_HIST_SNAPSHOT shwhere 1=1and st.SNAP_ID=sh.SNAP_IDand st.EVENT_NAME = 'enq: TX - row lock contention'order by sh.BEGIN_INTERVAL_TIME;
  • select s.BEGIN_INTERVAL_TIME,s.END_INTERVAL_TIME, t.VALUE EVALUE, LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) BVALUE,t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) DVALUE,s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME DTIME, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) H, EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) M, EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) S, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) Secs, (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/(EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)) VAL_SECfrom DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT twhere 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = 'parse count (hard)'order bys.BEGIN_INTERVAL_TIME;CREATE OR REPLACE function phy_get_secs(t_delta interval day to second)return number is sec_diff number;begin-- t_delta:= t2 - t1;sec_diff := ( extract(hour from t_delta)*60*60 + extract(minute from t_delta)*60 + extract(second from t_delta));return sec_diff;end;/show err
  • DBMS_XPLAN.DISPLAY_AWR
  • Oracle AWR Data mining

    1. 1. AWR Data miningYury Velikanov Senior Oracle DBA
    2. 2. Why Companies Trust Pythian • Recognized Leader: • Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and SQL Server • Work with over 150 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments • Expertise: • One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 6 Oracle ACEs/ACE Directors. • Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC. • Global Reach & Scalability: • 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response2 © 2011 Pythian
    3. 3. Mission Let you remember/consider AWR next time you troubleshoot Performance issue!3 © 2009/2010 Pythian
    4. 4. NOTE AWR = STATSPACK = Performance Repository Excerpt from 11GR2:$OH/rdbms/admin/awrrpt.sql “Rem This report is based on the Statspack report.”4 © 2009/2010 Pythian
    5. 5. AWR Agenda • Introduction & Background • Examples, Examples, Examples • Concept & Approach • More examples • Q&A Google: Oracle Yury Blog, Twitter, Linkedin, ACE … email, phone number5 © 2009/2010 Pythian
    6. 6. Few words about Yury • Google Yury Oracle [LinkedIn, twitter, blog, email, mobile, …] - Email me to get the presentation • Sr. Oracle DBA at Pythian, Oracle ACE and OCM • Started as Oracle DBA - with 7.2 (in 1997, 14+) • First international appearance - 2005 - Hotsos Symposium 2005 • Professional Education - Jonathan Lewis (2003 – 3 days), Tom Kyte (2004 – 3 days), Tanel Põder (2008 – 2 days ), Cary Millsap (2011 – 3 days) … • Education (Master Degree in Computer science) - OCP 7/8/8i/9/10/11 + OCM 9i/10g/11g • Oracle DBA consultant experience (14+ years) • Pythian Oracle Clients support (2+ years) - 140+ Clients around the world - Different products, different load, different problems6 © 2009/2010 Pythian
    7. 7. Background • AWR is one of many RDBMS performance data sources Jonathan Lewis / Tom Kyte / Tanel Põder / Cary Millsap • Sometimes it isn’t the best source (aggregation) • SQL Extended trace (event 10046) • RAW trace • tkprof • TRCAnlzr [ID 224270.1] • Method-R state of art tools • PL/SQL Profiler • LTOM (Session Trace Collector) • others • Sometimes it is the best source! • Sometimes it is the only one available!7 © 2009/2010 Pythian
    8. 8. Background • Once I was called to troubleshoot high load • Connected to the database I saw 8 active processes running for 6 hours in average at the time I connected • I switched 10046 event for all 8 collected 15 minutes data and analyzed it one by one • Found several SQLs returning 1 row million times • Passed the results to development asking to fix the logic • Spent ~2 hours to find where the issue was • Next day a workmate asked me • Why did you use 10046 and spent 2 hours? • He used AWR report and came up with the same answer in less than 5 minutes • Lesson learned: Right tool for the right (job - no) case !8 © 2009/2010 Pythian
    9. 9. When should you consider AWR mining? • General resource tuning (high CPU, IO utilization) • You are asked to reduce server load X times • You would like to analyze load patterns/trends • You need to go back in time and see how things progressed • You don’t have any other source of information • Existing official AWR interface doesn’t provide you information at the right angle/dimension or are not available (Grid Control, awrrpt.sql) • AWR SQL Execution Plans historical information analysis9 © 2009/2010 Pythian
    10. 10. TOP CPU/IO Consuming SQLs ? select s.SQL_ID, sum(CPU_TIME_DELTA), sum(DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT group by SQL_ID order by sum(CPU_TIME_DELTA) desc / SQL_ID SUM(CPU_TIME_DELTA) SUM(DISK_READS_DELTA) COUNT(*) ------------- ------------------- --------------------- ---------- 05s9358mm6vrr 27687500 2940 1 f6cz4n8y72xdc 7828125 4695 2 5dfmd823r8dsp 6421875 8 15 3h1rjtcff3wy1 5640625 113 1 92mb1kvurwn8h 5296875 0 1 bunssq950snhf 3937500 18 15 7xa8wfych4mad 2859375 0 2 ...10 © 2009/2010 Pythian
    11. 11. TOP CPU Consuming SQLs ? select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc11 © 2009/2010 Pythian
    12. 12. TOP CPU Consuming SQLs ? select * from ( select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc ) where rownum < 11 /12 © 2009/2010 Pythian
    13. 13. TOP CPU Consuming SQLs ? select * from ( select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p where 1=1 and s.SNAP_ID = p.SNAP_ID and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16 group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc ) where rownum < 11 /13 © 2009/2010 Pythian
    14. 14. TOP CPU Consuming SQLs ? select * from ( select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p where 1=1 and s.SNAP_ID = p.SNAP_ID and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16 and p.END_INTERVAL_TIME between SYSDATE-7 and SYSDATE group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc ) where rownum < 11 /14 © 2009/2010 Pythian
    15. 15. TOP CPU Consuming SQLs ? select * from ( select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p, DBA_HIST_SQLTEXT t where 1=1 and s.SNAP_ID = p.SNAP_ID and s.SQL_ID = t.SQL_ID and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16 and t.COMMAND_TYPE != 47 –- Exclude PL/SQL blocks from output and p.END_INTERVAL_TIME between SYSDATE-7 and SYSDATE group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc ) where rownum < 11 /15 © 2009/2010 Pythian
    16. 16. TOP CPU Consuming SQLs ? 2. 3. 5. 1. 52.8 % 4.16 © 2009/2010 Pythian
    17. 17. TOP CPU Consuming SQLs ? select SQL_ID, sum(CPU_TIME_DELTA), sum(DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT group by SQL_ID order by sum(CPU_TIME_DELTA) desc / SQL_ID SUM(CPU_TIME_DELTA) SUM(DISK_READS_DELTA) COUNT(*) ------------- ------------------- --------------------- ---------- 05s9358mm6vrr 27687500 2940 1 f6cz4n8y72xdc 7828125 4695 2 5dfmd823r8dsp 6421875 8 15 3h1rjtcff3wy1 5640625 113 1 92mb1kvurwn8h 5296875 0 1 bunssq950snhf 3937500 18 15 7xa8wfych4mad 2859375 0 2 ...17 © 2009/2010 Pythian
    18. 18. 5 Slides of Concept & Approach18 © 2009/2010 Pythian
    19. 19. AWR = DBA_HIST_% Views + • 111 Views in 11.2.0.2.0 • I use just few on a regular basis • DBA_HIST_ACTIVE_SESS_HISTORY - V$ACTIVE_SESSION_HISTORY • DBA_HIST_SEG_STAT - V$SEGMENT_STATISTICS • DBA_HIST_SQLSTAT - V$SQL • DBA_HIST_SQL_PLAN - V$SQL_PLAN • DBA_HIST_SYSSTAT - V$SYSSTAT ( ~SES~ ) • DBA_HIST_SYSTEM_EVENT - V$SYSTEM_EVENT ( ~SESSION~ ) • Most of the views contain data snapshots from V$___ views • DELTA columns (e.g. DISK_READS_DELTA) • DBA_HIST_SEG_STAT • DBA_HIST_SQLSTAT19 © 2009/2010 Pythian
    20. 20. AWR Things to keep in mind … • The data are just snapshots of V$ views • Data collected based on thresholds • Some data is excluded based on thresholds • Some data may not be in SGA at the time of snapshot • Longer time difference between snapshots more data got excluded • For data mining use ALL snapshots available Begin End t20 © 2009/2010 Pythian
    21. 21. AWR Things to keep in mind … • Forget about AWR if there are constants in the code • Indicator is high parse count (hard) (10-50 per/sec) • It isn’t just hard parsing! (related bugs) • cursor_sharing = FORCE • In RAC configuration do not forget INST_ID column in joins • Most of the V$ (DBA_HIST) performance views have incremental counters. END - BEGIN values • You may get wrong results (sometimes negative) • Sometimes counters reach max value and get reset • Counters got reset at instance restart time • Time between snapshots may be different • Suggestion (ENDv - BEGINv)/(ENDs - BEGINs)=value/sec21 © 2009/2010 Pythian
    22. 22. 22 -800000 -600000 -400000 -200000 200000 400000 600000 800000 0© 2009/2010 Pythian 150 200 250 100 50 0 2011.10.22… 2011.10.22… AWR Things to keep in mind … 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22…
    23. 23. AWR Things to keep in mind … • Seconds count between 2 timestamps select s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME, s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME DTIME, -- Returns “Interval” EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) H, EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) M, EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) S, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+ EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+ EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) SECS, phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) -– Write you own fun() (cast(s.END_INTERVAL_TIME as date) - cast(s.BEGIN_INTERVAL_TIME as date)) *24*60*60 from DBA_HIST_SNAPSHOT s where 1=1 and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) order by s.BEGIN_INTERVAL_TIME;23 © 2009/2010 Pythian
    24. 24. AWR Things to keep in mind … select SNAP_INTERVAL, RETENTION from DBA_HIST_WR_CONTROL c, V$DATABASE d where c.DBID = d.DBID; SNAP_INTERVAL RETENTION ------------------------------ ------------------------------ +00000 01:00:00.0 +00007 00:00:00.0 select DBID, INSTANCE_NUMBER, count(*) C, min(BEGIN_INTERVAL_TIME) OLDEST, max(BEGIN_INTERVAL_TIME) YUNGEST from DBA_HIST_SNAPSHOT group by DBID, INSTANCE_NUMBER; DBID INSTANCE_NUMBER C OLDEST YUNGEST ---------- --------------- ---------- ------------------------- ------------------------- 3244685755 1 179 13-AUG-11 07.00.30.233 PM 21-AUG-11 05.00.01.855 AM 3244685755 2 179 13-AUG-11 07.00.30.309 PM 21-AUG-11 05.00.01.761 AM24 © 2009/2010 Pythian
    25. 25. Trends Analysis Example (1) … DBA_HIST_SYSSTAT & DBA_HIST_SYSTEM_EVENT select s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME, ( t.VALUE- LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) ) DVALUE, (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/ phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) VAL_SEC from DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT t where 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = parse count (hard) order by s.BEGIN_INTERVAL_TIME;25 © 2009/2010 Pythian
    26. 26. Trends Analysis Example (1) …26 © 2009/2010 Pythian
    27. 27. Trends Analysis Example (1) … DBA_HIST_SYSSTAT & DBA_HIST_SYSTEM_EVENT select s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME, ( t.VALUE- LAG (t.VALUE) OVER (ORDER BY s.END_INTERVAL_TIME) ) DVALUE, (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.END_INTERVAL_TIME))/ phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) VAL_SEC from DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT t where 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = parse count (hard) order by s.END_INTERVAL_TIME;27 © 2009/2010 Pythian
    28. 28. SQL Bad performance Example (2) … • Called by a client to troubleshoot a SQL with bad performance • Sometimes the SQL hangs (never finishes) and needs to be killed and re-executed • Upon re-execution, it always finishes successfully in a few minutes • The client demanded a resolution ASAP …28 © 2009/2010 Pythian
    29. 29. SQL Bad performance Example (2) … DBA_HIST_SQLSTAT select st.SQL_ID , st.PLAN_HASH_VALUE , sum(st.EXECUTIONS_DELTA) EXECUTIONS , sum(st.ROWS_PROCESSED_DELTA) CROWS , trunc(sum(st.CPU_TIME_DELTA)/1000000/60) CPU_MINS , trunc(sum(st.ELAPSED_TIME_DELTA)/1000000/60) ELA_MINS from DBA_HIST_SQLSTAT st where st.SQL_ID in ( 5ppdcygtcw7p6 ,gpj32cqd0qy9a ) group by st.SQL_ID , st.PLAN_HASH_VALUE order by st.SQL_ID, CPU_MINS;29 © 2009/2010 Pythian
    30. 30. SQL Bad performance Example (2) … DBA_HIST_SQLSTAT SQL_ID PLAN_HASH_VALUE EXECUTIONS CROWS CPU_MINS ELA_MINS ------------- --------------- ---------- ---------- ---------------- ---------------- 5ppdcygtcw7p6 436796090 20 82733 1 3 5ppdcygtcw7p6 863350916 71 478268 5 11 5ppdcygtcw7p6 2817686509 9 32278 2,557 2,765 gpj32cqd0qy9a 3094138997 30 58400 1 3 gpj32cqd0qy9a 1700210966 36 69973 1 7 gpj32cqd0qy9a 1168845432 2 441 482 554 gpj32cqd0qy9a 2667660534 4 1489 1,501 1,64230 © 2009/2010 Pythian
    31. 31. SQL Bad performance Example (2) … DBA_HIST_SQLSTAT & DBA_HIST_SEG_STAT select st.SQL_ID , st.PLAN_HASH_VALUE , sum(st.EXECUTIONS_DELTA) EXECUTIONS , sum(st.ROWS_PROCESSED_DELTA) CROWS , trunc(sum(st.CPU_TIME_DELTA)/1000000/60) CPU_MINS , trunc(sum(st.ELAPSED_TIME_DELTA)/1000000/60) ELA_MINS from DBA_HIST_SQLSTAT st where st.SQL_ID in ( 5ppdcygtcw7p6 ,gpj32cqd0qy9a ) group by st.SQL_ID , st.PLAN_HASH_VALUE order by st.SQL_ID, CPU_MINS;31 © 2009/2010 Pythian
    32. 32. SQL Bad performance Example (2) … • In the result … • Two different jobs were gathering statistics on a daily basis 1. “ANALYZE …” part of other batch job (developer) 2. “DBMS_STATS…” traditional (DBA) • Sometimes “DBMS_STATS…“ is not completed before the batch job starts (+/- 10 minutes). • After the job got killed (typically well after 10 mins since start) the new “correct” statistics were in place. • Takeaways … A. Don’t change your statistics that frequently (should be consistent) B. AWR data helps to spot such issues easily32 © 2009/2010 Pythian
    33. 33. SQL Plan flipping Example (3) … • I asked myself: Well ! • If you find that the execution plan for a SQL has changed from a good (quick) to a bad one (slow), how do you know if there are other affected SQLs? • And if there are, how many and which ones? • Would SQL Profiles (Outlines) help address those?33 © 2009/2010 Pythian
    34. 34. SQL Plan flipping Example (3) … SELECT st2.SQL_ID , st2.PLAN_HASH_VALUE , st_long.PLAN_HASH_VALUE l_PLAN_HASH_VALUE , st2.CPU_MINS , st_long.CPU_MINS l_CPU_MINS , st2.ELA_MINS , st_long.ELA_MINS l_ELA_MINS , st2.EXECUTIONS , st_long.EXECUTIONS l_EXECUTIONS , st2.CROWS , st_long.CROWS l_CROWS , st2.CPU_MINS_PER_ROW , st_long.CPU_MINS_PER_ROW l_CPU_MINS_PER_ROW FROM (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , SUM(st.EXECUTIONS_DELTA) EXECUTIONS , SUM(st.ROWS_PROCESSED_DELTA) CROWS , TRUNC(SUM(st.CPU_TIME_DELTA) /1000000/60) CPU_MINS , DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW , TRUNC(SUM(st.ELAPSED_TIME_DELTA) /1000000/60) ELA_MINS FROM DBA_HIST_SQLSTAT st WHERE 1 =1 AND ( st.CPU_TIME_DELTA !=0 OR st.ROWS_PROCESSED_DELTA !=0) GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st2, (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , SUM(st.EXECUTIONS_DELTA) EXECUTIONS , SUM(st.ROWS_PROCESSED_DELTA) CROWS , TRUNC(SUM(st.CPU_TIME_DELTA) /1000000/60) CPU_MINS , DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW , TRUNC(SUM(st.ELAPSED_TIME_DELTA) /1000000/60) ELA_MINS FROM DBA_HIST_SQLSTAT st WHERE 1 =1 AND ( st.CPU_TIME_DELTA !=0 OR st.ROWS_PROCESSED_DELTA !=0) HAVING TRUNC(SUM(st.CPU_TIME_DELTA)/1000000/60) > 10 GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st_long WHERE 1 =1 AND st2.SQL_ID = st_long.SQL_ID AND st_long.CPU_MINS_PER_ROW/DECODE(st2.CPU_MINS_PER_ROW,0,1,st2.CPU_MINS_PER_ROW) > 2 ORDER BY l_CPU_MINS DESC, st2.SQL_ID, st_long.CPU_MINS DESC, st2.PLAN_HASH_VALUE;34 © 2009/2010 Pythian
    35. 35. SQL Plan flipping Example (3) … SELECT ... FROM (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , ... DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW , ... FROM DBA_HIST_SQLSTAT st WHERE 1 =1 ... GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st2, (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , ... HAVING trunc(sum(st.CPU_TIME_DELTA)/1000000/60) > 10 GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st_long WHERE 1 =1 AND st2.SQL_ID = st_long.SQL_ID AND st_long.CPU_MINS_PER_ROW/DECODE(st2.CPU_MINS_PER_ROW,0,1,st2.CPU_MINS_PER_ROW) > 2 ORDER BY l_CPU_MINS DESC, st2.SQL_ID, st_long.CPU_MINS DESC, st2.PLAN_HASH_VALUE;35 © 2009/2010 Pythian
    36. 36. SQL Plan flipping Example (3) …SQL_ID PLAN_HASH_VALUE L_PLAN_HASH_VALUE CPU_MINS L_CPU_MINS ELA_MINS L_ELA_MINS EXECUTIONS L_EXECUTIONS------------- --------------- ----------------- ---------- ---------- ---------- ---------- ---------- ------------db8yz0rfhvufm 3387634876 619162475 17 2673 21 4074 3106638 1935ppdcygtcw7p6 436796090 2817686509 1 2557 3 2765 20 95ppdcygtcw7p6 863350916 2817686509 5 2557 11 2765 71 91tab7mjut8j9h 875484785 911605088 9 2112 23 2284 980 14361tab7mjut8j9h 2484900321 911605088 6 2112 6 2284 1912 14361tab7mjut8j9h 3141038411 911605088 50 2112 57 2284 32117 1436gpj32cqd0qy9a 1700210966 2667660534 1 1501 7 1642 36 4gpj32cqd0qy9a 3094138997 2667660534 1 1501 3 1642 30 42tf4p2anpwpk2 825403357 1679851684 6 824 71 913 17 13csvwu3kqu43j4 3860135778 2851322291 0 784 0 874 1 20q9hpmtk8c1hf 3860135778 2851322291 0 779 0 867 1 22frwhbxvg1j69 3860135778 2851322291 0 776 0 865 1 24nzsxm3d9rspt 3860135778 2851322291 0 754 0 846 1 21pc2npdb1kbp6 9772089 2800812079 0 511 0 3000 7 695gpj32cqd0qy9a 1700210966 1168845432 1 482 7 554 36 2gpj32cqd0qy9a 3094138997 1168845432 1 482 3 554 30 2 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *4bcx6kbbrg6bv 3781789023 2248191382 0 11 0 41 2 26wh3untj05apd 3457450300 3233890669 0 11 0 131 1 206wh3untj05apd 3477405755 3233890669 0 11 1 131 2 208pzsjt5p64xfu 3998876049 3667423051 0 11 5 44 3 18bpfzx2hxf5x7f 1890295626 774548604 0 11 0 26 1 24g67nkxd2nqqqd 1308088852 4202046543 0 11 1 57 1 49g67nkxd2nqqqd 1308088852 1991738870 0 11 1 39 1 38g67nkxd2nqqqd 2154937993 1991738870 1 11 27 39 72 38g67nkxd2nqqqd 2154937993 4202046543 1 11 27 57 72 4992 rows selected.Elapsed: 00:00:02.53SQL>36 © 2009/2010 Pythian
    37. 37. SQL Plan flipping Example (3) … • In the result … • Load on the system was reduced by 5 times • Takeaways … A. SQL Plans may flip from good plans to … B. SQL Outlines/Profiles may help some times C. AWR provides good input for such analysis • Why SQL Plans may flip? 1. Bind variable peeking / adaptive cursor sharing 2. Statistics change (including difference in partitions stats) 3. Adding/Removing indexes 4. Session/System init.ora parameters (nls_sort/optimizer_mode) 5. Dynamic statistics gathering (sampling) 6. Profiles/Outlines/Baselines evolution37 © 2009/2010 Pythian
    38. 38. Conclusions … • AWR = DBA_HIST% views ( snapshots from V$% views ) • Sometimes it is the only source of information • AWR contains much more information that default AWR reports and Grid Control could provide you • Be careful mining data (there are some gotchas) • Don’t be afraid to discover/mine the AWR data I can show you the door … … but it is you who should walk through it38 © 2009/2010 Pythian
    39. 39. Pythian Facts• Founded in 1997, over 14 years• 100+ DBAs ! (140 employees)• 5 offices in 5 countries (true follow the sun model)• Employ • 6 Oracle ACEs (Including 1 ACE director) • Several Oracle Masters • Plenty of technical geeks• Platinum level partner in the Oracle Partner Network• Actively supports technical communities via • Blogging • Conferences • SIGs and other events39 © 2009/2010 Pythian
    40. 40. Google Yury Oracle [LinkedIn, twitter, blog, email, mobile, …]Additionalget the presentation Email me to Resources • www.oracle.com/scan • www.pythian.com/exadata • www.pythian.com/news/tag/exadata - Exadata Mission Blog Let you remember/consider AWR • www.pythian.com/news_and_events/in_the_newsArticle: “Making the Most time youExadata” next of Oracle troubleshoot Performance issue!My Oracle Support notes 888828.1 and 757552.1Thank you! Pythian •Like Pythian on facebook: http://on.fb.me/pythianfacebook •Follow us on LinkedIn: http://linkd.in/pythian •Follow Pythian on Twitter @pythian (www.twitter.com/pythian)40 © 2009/2010 Pythian

    ×