• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Oracle AWR Data mining
 

Oracle AWR Data mining

on

  • 5,023 views

The presentation on how to mine Oracle Automatic Workload Repository.

The presentation on how to mine Oracle Automatic Workload Repository.

Statistics

Views

Total Views
5,023
Views on SlideShare
4,969
Embed Views
54

Actions

Likes
4
Downloads
398
Comments
1

5 Embeds 54

http://www.linkedin.com 40
http://www.slashdocs.com 7
https://www.linkedin.com 3
http://www.docshut.com 2
https://twitter.com 2

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel

11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • Hi Yury,

    can we get audio of this presentation?

    Thanks,
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Trace Analyzer, also known as TRCANLZR or TRCA, is a tool provided by Oracle Server Technologies Center of Expertise - ST CoE. TRCA inputs one or several SQL trace(s) generated by Event 10046 and outputs a diagnostics report in two formats (html and text). These reports are commonly used to diagnose processes performing poorly.
  • REM File: yury1.sqlset echo off feed off lines 110 pages 9990REM Copyright (c) 2011, TPG, all rights reservedclear breakclear colclear computettitle offbtitle offundefine starthh24undefine endhh24undefinedaysagocol beginttim heading 'Begin|Interval' format a15 trunccol numtimes format 9,990 head 'Num|Times'col sql_id heading 'SQL ID' format a13col sumcputimeDelta heading 'SUM|CPU|Time|Delta' format 9,999,999,990col buffer_gets_Delta heading 'Buff|Gets' format 999,999,990col sumdiskreadsDelta heading 'SUM|Disk|Reads' format 9,999,990col sumiowaitDelta heading 'SUM|IO|Wait' format 9,999,999,990break on beginttim skip 1ttitle skip 1 center 'Top 10 SQL Delta Statistics &&daysago Days ago Hours &&starthh24 to &&endhh24' skip 2spool yury1.lisselect * from( selects.SQL_ID , sum( s.CPU_TIME_DELTA ) sumCPUTIMEDELTA , sum( s.DISK_READS_DELTA ) sumDISKREADSDELTA , sum( s.IOWAIT_DELTA ) sumiowaitDELTA , count( * ) numtimes from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p, DBA_HIST_SQLTEXT t where ( s.SNAP_ID = p.SNAP_ID ) and ( s.SQL_ID = t.SQL_ID ) and ( EXTRACT( HOUR FROM p.END_INTERVAL_TIME ) between &starthh24 and &endhh24 ) and ( t.COMMAND_TYPE != 47 ) -- Exclude PL/SQL blocks from output and ( p.END_INTERVAL_TIME between ( SYSDATE - &daysago ) and SYSDATE ) group bys.SQL_ID order by sum( s.CPU_TIME_DELTA ) desc )whererownum < 11/PROMPTspool offclear breakclear colclear computettitle offbtitle offset echo on feedback on lines 80 pages 60 term on
  • CREATE OR REPLACE function phy_get_secs (t1 in timestamp, t2 in timestamp)return number is sec_diff number;t_delta interval day to second;begint_delta := t2 - t1;sec_diff := ( extract(hour from t_delta)*60*60 + extract(minute from t_delta)*60 + extract(second from t_delta));return sec_diff;end;/show err
  • set pages 1000prompt SNAP_TIME_P|SNAP_TIME|SEC_D|VALUE_P|VALUE|VALUE_D|VALUE_PER_SECselect /*+ ORDERED FULL(st) */ LAG (st.SNAP_TIME) OVER (ORDER BY st.SNAP_TIME)||'|'||st.SNAP_TIME||'|'||trunc((st.SNAP_TIME - LAG (st.SNAP_TIME) OVER (ORDER BY st.SNAP_TIME))*24*60*60)||'|'|| LAG (ss.value) OVER (ORDER BY st.SNAP_TIME)||'|'||ss.value||'|'|| ( ss.value - LAG (ss.value) OVER (ORDER BY st.SNAP_TIME) )||'|'||trunc( ( ss.value - LAG (ss.value) OVER (ORDER BY st.SNAP_TIME) ) / ((st.SNAP_TIME - LAG (st.SNAP_TIME) OVER (ORDER BY st.SNAP_TIME))*24*60*60) ) fromv$databasevd,stats$snapshotst, V$STATNAME sn, STATS$SYSSTAT ss where 1=1 and ss.snap_id=st.snap_id and ss.instance_number=1 and ss.dbid=vd.dbid and sn.STATISTIC#=ss.STATISTIC# and sn.name='parse count (hard)' and st.SNAP_TIME between trunc(sysdate)-14 and trunc(sysdate) order by st.SNAP_TIME;select s.BEGIN_INTERVAL_TIME,s.END_INTERVAL_TIME, t.VALUE EVALUE, LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) BVALUE,t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) DVALUE,s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME DTIME, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) H, EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) M, EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) S, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) Secs, (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/(EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)) VAL_SECfrom DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT twhere 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = 'parse count (hard)'order bys.BEGIN_INTERVAL_TIME;CREATE OR REPLACE function phy_get_secs (t1 in timestamp, t2 in timestamp)return number is sec_diff number;t_delta interval day to second;begint_delta := t2 - t1;sec_diff := ( extract(hour from t_delta)*60*60 + extract(minute from t_delta)*60 + extract(second from t_delta));return sec_diff;end;/show errcolumn snap_date for a20select cast (s.BEGIN_INTERVAL_TIME as date) snap_date, (t.TIME_WAITED_MICRO-LAG (t.TIME_WAITED_MICRO) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/(t.TOTAL_WAITS-LAG (t.TOTAL_WAITS) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/1000 ms_per_ev, (t.TOTAL_WAITS-LAG (t.TOTAL_WAITS) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/(EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)) ev_SECfrom DBA_HIST_SNAPSHOT s, DBA_HIST_SYSTEM_EVENT twhere 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.EVENT_NAME = 'db file scattered read'order bys.BEGIN_INTERVAL_TIME;select sh.BEGIN_INTERVAL_TIME d ,trunc(st.TOTAL_WAITS-(LAG(st.TOTAL_WAITS) OVER (ORDER BY sh.BEGIN_INTERVAL_TIME))) tw_d ,trunc(st.TOTAL_TIMEOUTS-(LAG(st.TOTAL_TIMEOUTS) OVER (ORDER BY sh.BEGIN_INTERVAL_TIME))) tt_d ,trunc((st.TIME_WAITED_MICRO-(LAG(st.TIME_WAITED_MICRO) OVER (ORDER BY sh.BEGIN_INTERVAL_TIME)))/1000000) w_d_secsfrom DBA_HIST_SYSTEM_EVENT st, DBA_HIST_SNAPSHOT shwhere 1=1and st.SNAP_ID=sh.SNAP_IDand st.EVENT_NAME = 'enq: TX - row lock contention'order by sh.BEGIN_INTERVAL_TIME;
  • select s.BEGIN_INTERVAL_TIME,s.END_INTERVAL_TIME, t.VALUE EVALUE, LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) BVALUE,t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) DVALUE,s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME DTIME, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) H, EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) M, EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) S, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) Secs, (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/(EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)) VAL_SECfrom DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT twhere 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = 'parse count (hard)'order bys.BEGIN_INTERVAL_TIME;CREATE OR REPLACE function phy_get_secs(t_delta interval day to second)return number is sec_diff number;begin-- t_delta:= t2 - t1;sec_diff := ( extract(hour from t_delta)*60*60 + extract(minute from t_delta)*60 + extract(second from t_delta));return sec_diff;end;/show err
  • DBMS_XPLAN.DISPLAY_AWR

Oracle AWR Data mining Oracle AWR Data mining Presentation Transcript

  • AWR Data miningYury Velikanov Senior Oracle DBA
  • Why Companies Trust Pythian • Recognized Leader: • Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and SQL Server • Work with over 150 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments • Expertise: • One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 6 Oracle ACEs/ACE Directors. • Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC. • Global Reach & Scalability: • 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response2 © 2011 Pythian
  • Mission Let you remember/consider AWR next time you troubleshoot Performance issue!3 © 2009/2010 Pythian
  • NOTE AWR = STATSPACK = Performance Repository Excerpt from 11GR2:$OH/rdbms/admin/awrrpt.sql “Rem This report is based on the Statspack report.”4 © 2009/2010 Pythian
  • AWR Agenda • Introduction & Background • Examples, Examples, Examples • Concept & Approach • More examples • Q&A Google: Oracle Yury Blog, Twitter, Linkedin, ACE … email, phone number5 © 2009/2010 Pythian
  • Few words about Yury • Google Yury Oracle [LinkedIn, twitter, blog, email, mobile, …] - Email me to get the presentation • Sr. Oracle DBA at Pythian, Oracle ACE and OCM • Started as Oracle DBA - with 7.2 (in 1997, 14+) • First international appearance - 2005 - Hotsos Symposium 2005 • Professional Education - Jonathan Lewis (2003 – 3 days), Tom Kyte (2004 – 3 days), Tanel Põder (2008 – 2 days ), Cary Millsap (2011 – 3 days) … • Education (Master Degree in Computer science) - OCP 7/8/8i/9/10/11 + OCM 9i/10g/11g • Oracle DBA consultant experience (14+ years) • Pythian Oracle Clients support (2+ years) - 140+ Clients around the world - Different products, different load, different problems6 © 2009/2010 Pythian
  • Background • AWR is one of many RDBMS performance data sources Jonathan Lewis / Tom Kyte / Tanel Põder / Cary Millsap • Sometimes it isn’t the best source (aggregation) • SQL Extended trace (event 10046) • RAW trace • tkprof • TRCAnlzr [ID 224270.1] • Method-R state of art tools • PL/SQL Profiler • LTOM (Session Trace Collector) • others • Sometimes it is the best source! • Sometimes it is the only one available!7 © 2009/2010 Pythian
  • Background • Once I was called to troubleshoot high load • Connected to the database I saw 8 active processes running for 6 hours in average at the time I connected • I switched 10046 event for all 8 collected 15 minutes data and analyzed it one by one • Found several SQLs returning 1 row million times • Passed the results to development asking to fix the logic • Spent ~2 hours to find where the issue was • Next day a workmate asked me • Why did you use 10046 and spent 2 hours? • He used AWR report and came up with the same answer in less than 5 minutes • Lesson learned: Right tool for the right (job - no) case !8 © 2009/2010 Pythian
  • When should you consider AWR mining? • General resource tuning (high CPU, IO utilization) • You are asked to reduce server load X times • You would like to analyze load patterns/trends • You need to go back in time and see how things progressed • You don’t have any other source of information • Existing official AWR interface doesn’t provide you information at the right angle/dimension or are not available (Grid Control, awrrpt.sql) • AWR SQL Execution Plans historical information analysis9 © 2009/2010 Pythian
  • TOP CPU/IO Consuming SQLs ? select s.SQL_ID, sum(CPU_TIME_DELTA), sum(DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT group by SQL_ID order by sum(CPU_TIME_DELTA) desc / SQL_ID SUM(CPU_TIME_DELTA) SUM(DISK_READS_DELTA) COUNT(*) ------------- ------------------- --------------------- ---------- 05s9358mm6vrr 27687500 2940 1 f6cz4n8y72xdc 7828125 4695 2 5dfmd823r8dsp 6421875 8 15 3h1rjtcff3wy1 5640625 113 1 92mb1kvurwn8h 5296875 0 1 bunssq950snhf 3937500 18 15 7xa8wfych4mad 2859375 0 2 ...10 © 2009/2010 Pythian
  • TOP CPU Consuming SQLs ? select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc11 © 2009/2010 Pythian
  • TOP CPU Consuming SQLs ? select * from ( select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc ) where rownum < 11 /12 © 2009/2010 Pythian
  • TOP CPU Consuming SQLs ? select * from ( select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p where 1=1 and s.SNAP_ID = p.SNAP_ID and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16 group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc ) where rownum < 11 /13 © 2009/2010 Pythian
  • TOP CPU Consuming SQLs ? select * from ( select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p where 1=1 and s.SNAP_ID = p.SNAP_ID and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16 and p.END_INTERVAL_TIME between SYSDATE-7 and SYSDATE group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc ) where rownum < 11 /14 © 2009/2010 Pythian
  • TOP CPU Consuming SQLs ? select * from ( select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p, DBA_HIST_SQLTEXT t where 1=1 and s.SNAP_ID = p.SNAP_ID and s.SQL_ID = t.SQL_ID and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16 and t.COMMAND_TYPE != 47 –- Exclude PL/SQL blocks from output and p.END_INTERVAL_TIME between SYSDATE-7 and SYSDATE group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc ) where rownum < 11 /15 © 2009/2010 Pythian
  • TOP CPU Consuming SQLs ? 2. 3. 5. 1. 52.8 % 4.16 © 2009/2010 Pythian
  • TOP CPU Consuming SQLs ? select SQL_ID, sum(CPU_TIME_DELTA), sum(DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT group by SQL_ID order by sum(CPU_TIME_DELTA) desc / SQL_ID SUM(CPU_TIME_DELTA) SUM(DISK_READS_DELTA) COUNT(*) ------------- ------------------- --------------------- ---------- 05s9358mm6vrr 27687500 2940 1 f6cz4n8y72xdc 7828125 4695 2 5dfmd823r8dsp 6421875 8 15 3h1rjtcff3wy1 5640625 113 1 92mb1kvurwn8h 5296875 0 1 bunssq950snhf 3937500 18 15 7xa8wfych4mad 2859375 0 2 ...17 © 2009/2010 Pythian
  • 5 Slides of Concept & Approach18 © 2009/2010 Pythian
  • AWR = DBA_HIST_% Views + • 111 Views in 11.2.0.2.0 • I use just few on a regular basis • DBA_HIST_ACTIVE_SESS_HISTORY - V$ACTIVE_SESSION_HISTORY • DBA_HIST_SEG_STAT - V$SEGMENT_STATISTICS • DBA_HIST_SQLSTAT - V$SQL • DBA_HIST_SQL_PLAN - V$SQL_PLAN • DBA_HIST_SYSSTAT - V$SYSSTAT ( ~SES~ ) • DBA_HIST_SYSTEM_EVENT - V$SYSTEM_EVENT ( ~SESSION~ ) • Most of the views contain data snapshots from V$___ views • DELTA columns (e.g. DISK_READS_DELTA) • DBA_HIST_SEG_STAT • DBA_HIST_SQLSTAT19 © 2009/2010 Pythian
  • AWR Things to keep in mind … • The data are just snapshots of V$ views • Data collected based on thresholds • Some data is excluded based on thresholds • Some data may not be in SGA at the time of snapshot • Longer time difference between snapshots more data got excluded • For data mining use ALL snapshots available Begin End t20 © 2009/2010 Pythian
  • AWR Things to keep in mind … • Forget about AWR if there are constants in the code • Indicator is high parse count (hard) (10-50 per/sec) • It isn’t just hard parsing! (related bugs) • cursor_sharing = FORCE • In RAC configuration do not forget INST_ID column in joins • Most of the V$ (DBA_HIST) performance views have incremental counters. END - BEGIN values • You may get wrong results (sometimes negative) • Sometimes counters reach max value and get reset • Counters got reset at instance restart time • Time between snapshots may be different • Suggestion (ENDv - BEGINv)/(ENDs - BEGINs)=value/sec21 © 2009/2010 Pythian
  • 22 -800000 -600000 -400000 -200000 200000 400000 600000 800000 0© 2009/2010 Pythian 150 200 250 100 50 0 2011.10.22… 2011.10.22… AWR Things to keep in mind … 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22…
  • AWR Things to keep in mind … • Seconds count between 2 timestamps select s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME, s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME DTIME, -- Returns “Interval” EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) H, EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) M, EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) S, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+ EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+ EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) SECS, phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) -– Write you own fun() (cast(s.END_INTERVAL_TIME as date) - cast(s.BEGIN_INTERVAL_TIME as date)) *24*60*60 from DBA_HIST_SNAPSHOT s where 1=1 and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) order by s.BEGIN_INTERVAL_TIME;23 © 2009/2010 Pythian
  • AWR Things to keep in mind … select SNAP_INTERVAL, RETENTION from DBA_HIST_WR_CONTROL c, V$DATABASE d where c.DBID = d.DBID; SNAP_INTERVAL RETENTION ------------------------------ ------------------------------ +00000 01:00:00.0 +00007 00:00:00.0 select DBID, INSTANCE_NUMBER, count(*) C, min(BEGIN_INTERVAL_TIME) OLDEST, max(BEGIN_INTERVAL_TIME) YUNGEST from DBA_HIST_SNAPSHOT group by DBID, INSTANCE_NUMBER; DBID INSTANCE_NUMBER C OLDEST YUNGEST ---------- --------------- ---------- ------------------------- ------------------------- 3244685755 1 179 13-AUG-11 07.00.30.233 PM 21-AUG-11 05.00.01.855 AM 3244685755 2 179 13-AUG-11 07.00.30.309 PM 21-AUG-11 05.00.01.761 AM24 © 2009/2010 Pythian
  • Trends Analysis Example (1) … DBA_HIST_SYSSTAT & DBA_HIST_SYSTEM_EVENT select s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME, ( t.VALUE- LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) ) DVALUE, (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/ phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) VAL_SEC from DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT t where 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = parse count (hard) order by s.BEGIN_INTERVAL_TIME;25 © 2009/2010 Pythian
  • Trends Analysis Example (1) …26 © 2009/2010 Pythian
  • Trends Analysis Example (1) … DBA_HIST_SYSSTAT & DBA_HIST_SYSTEM_EVENT select s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME, ( t.VALUE- LAG (t.VALUE) OVER (ORDER BY s.END_INTERVAL_TIME) ) DVALUE, (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.END_INTERVAL_TIME))/ phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) VAL_SEC from DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT t where 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = parse count (hard) order by s.END_INTERVAL_TIME;27 © 2009/2010 Pythian
  • SQL Bad performance Example (2) … • Called by a client to troubleshoot a SQL with bad performance • Sometimes the SQL hangs (never finishes) and needs to be killed and re-executed • Upon re-execution, it always finishes successfully in a few minutes • The client demanded a resolution ASAP …28 © 2009/2010 Pythian
  • SQL Bad performance Example (2) … DBA_HIST_SQLSTAT select st.SQL_ID , st.PLAN_HASH_VALUE , sum(st.EXECUTIONS_DELTA) EXECUTIONS , sum(st.ROWS_PROCESSED_DELTA) CROWS , trunc(sum(st.CPU_TIME_DELTA)/1000000/60) CPU_MINS , trunc(sum(st.ELAPSED_TIME_DELTA)/1000000/60) ELA_MINS from DBA_HIST_SQLSTAT st where st.SQL_ID in ( 5ppdcygtcw7p6 ,gpj32cqd0qy9a ) group by st.SQL_ID , st.PLAN_HASH_VALUE order by st.SQL_ID, CPU_MINS;29 © 2009/2010 Pythian
  • SQL Bad performance Example (2) … DBA_HIST_SQLSTAT SQL_ID PLAN_HASH_VALUE EXECUTIONS CROWS CPU_MINS ELA_MINS ------------- --------------- ---------- ---------- ---------------- ---------------- 5ppdcygtcw7p6 436796090 20 82733 1 3 5ppdcygtcw7p6 863350916 71 478268 5 11 5ppdcygtcw7p6 2817686509 9 32278 2,557 2,765 gpj32cqd0qy9a 3094138997 30 58400 1 3 gpj32cqd0qy9a 1700210966 36 69973 1 7 gpj32cqd0qy9a 1168845432 2 441 482 554 gpj32cqd0qy9a 2667660534 4 1489 1,501 1,64230 © 2009/2010 Pythian
  • SQL Bad performance Example (2) … DBA_HIST_SQLSTAT & DBA_HIST_SEG_STAT select st.SQL_ID , st.PLAN_HASH_VALUE , sum(st.EXECUTIONS_DELTA) EXECUTIONS , sum(st.ROWS_PROCESSED_DELTA) CROWS , trunc(sum(st.CPU_TIME_DELTA)/1000000/60) CPU_MINS , trunc(sum(st.ELAPSED_TIME_DELTA)/1000000/60) ELA_MINS from DBA_HIST_SQLSTAT st where st.SQL_ID in ( 5ppdcygtcw7p6 ,gpj32cqd0qy9a ) group by st.SQL_ID , st.PLAN_HASH_VALUE order by st.SQL_ID, CPU_MINS;31 © 2009/2010 Pythian
  • SQL Bad performance Example (2) … • In the result … • Two different jobs were gathering statistics on a daily basis 1. “ANALYZE …” part of other batch job (developer) 2. “DBMS_STATS…” traditional (DBA) • Sometimes “DBMS_STATS…“ is not completed before the batch job starts (+/- 10 minutes). • After the job got killed (typically well after 10 mins since start) the new “correct” statistics were in place. • Takeaways … A. Don’t change your statistics that frequently (should be consistent) B. AWR data helps to spot such issues easily32 © 2009/2010 Pythian
  • SQL Plan flipping Example (3) … • I asked myself: Well ! • If you find that the execution plan for a SQL has changed from a good (quick) to a bad one (slow), how do you know if there are other affected SQLs? • And if there are, how many and which ones? • Would SQL Profiles (Outlines) help address those?33 © 2009/2010 Pythian
  • SQL Plan flipping Example (3) … SELECT st2.SQL_ID , st2.PLAN_HASH_VALUE , st_long.PLAN_HASH_VALUE l_PLAN_HASH_VALUE , st2.CPU_MINS , st_long.CPU_MINS l_CPU_MINS , st2.ELA_MINS , st_long.ELA_MINS l_ELA_MINS , st2.EXECUTIONS , st_long.EXECUTIONS l_EXECUTIONS , st2.CROWS , st_long.CROWS l_CROWS , st2.CPU_MINS_PER_ROW , st_long.CPU_MINS_PER_ROW l_CPU_MINS_PER_ROW FROM (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , SUM(st.EXECUTIONS_DELTA) EXECUTIONS , SUM(st.ROWS_PROCESSED_DELTA) CROWS , TRUNC(SUM(st.CPU_TIME_DELTA) /1000000/60) CPU_MINS , DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW , TRUNC(SUM(st.ELAPSED_TIME_DELTA) /1000000/60) ELA_MINS FROM DBA_HIST_SQLSTAT st WHERE 1 =1 AND ( st.CPU_TIME_DELTA !=0 OR st.ROWS_PROCESSED_DELTA !=0) GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st2, (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , SUM(st.EXECUTIONS_DELTA) EXECUTIONS , SUM(st.ROWS_PROCESSED_DELTA) CROWS , TRUNC(SUM(st.CPU_TIME_DELTA) /1000000/60) CPU_MINS , DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW , TRUNC(SUM(st.ELAPSED_TIME_DELTA) /1000000/60) ELA_MINS FROM DBA_HIST_SQLSTAT st WHERE 1 =1 AND ( st.CPU_TIME_DELTA !=0 OR st.ROWS_PROCESSED_DELTA !=0) HAVING TRUNC(SUM(st.CPU_TIME_DELTA)/1000000/60) > 10 GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st_long WHERE 1 =1 AND st2.SQL_ID = st_long.SQL_ID AND st_long.CPU_MINS_PER_ROW/DECODE(st2.CPU_MINS_PER_ROW,0,1,st2.CPU_MINS_PER_ROW) > 2 ORDER BY l_CPU_MINS DESC, st2.SQL_ID, st_long.CPU_MINS DESC, st2.PLAN_HASH_VALUE;34 © 2009/2010 Pythian
  • SQL Plan flipping Example (3) … SELECT ... FROM (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , ... DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW , ... FROM DBA_HIST_SQLSTAT st WHERE 1 =1 ... GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st2, (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , ... HAVING trunc(sum(st.CPU_TIME_DELTA)/1000000/60) > 10 GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st_long WHERE 1 =1 AND st2.SQL_ID = st_long.SQL_ID AND st_long.CPU_MINS_PER_ROW/DECODE(st2.CPU_MINS_PER_ROW,0,1,st2.CPU_MINS_PER_ROW) > 2 ORDER BY l_CPU_MINS DESC, st2.SQL_ID, st_long.CPU_MINS DESC, st2.PLAN_HASH_VALUE;35 © 2009/2010 Pythian
  • SQL Plan flipping Example (3) …SQL_ID PLAN_HASH_VALUE L_PLAN_HASH_VALUE CPU_MINS L_CPU_MINS ELA_MINS L_ELA_MINS EXECUTIONS L_EXECUTIONS------------- --------------- ----------------- ---------- ---------- ---------- ---------- ---------- ------------db8yz0rfhvufm 3387634876 619162475 17 2673 21 4074 3106638 1935ppdcygtcw7p6 436796090 2817686509 1 2557 3 2765 20 95ppdcygtcw7p6 863350916 2817686509 5 2557 11 2765 71 91tab7mjut8j9h 875484785 911605088 9 2112 23 2284 980 14361tab7mjut8j9h 2484900321 911605088 6 2112 6 2284 1912 14361tab7mjut8j9h 3141038411 911605088 50 2112 57 2284 32117 1436gpj32cqd0qy9a 1700210966 2667660534 1 1501 7 1642 36 4gpj32cqd0qy9a 3094138997 2667660534 1 1501 3 1642 30 42tf4p2anpwpk2 825403357 1679851684 6 824 71 913 17 13csvwu3kqu43j4 3860135778 2851322291 0 784 0 874 1 20q9hpmtk8c1hf 3860135778 2851322291 0 779 0 867 1 22frwhbxvg1j69 3860135778 2851322291 0 776 0 865 1 24nzsxm3d9rspt 3860135778 2851322291 0 754 0 846 1 21pc2npdb1kbp6 9772089 2800812079 0 511 0 3000 7 695gpj32cqd0qy9a 1700210966 1168845432 1 482 7 554 36 2gpj32cqd0qy9a 3094138997 1168845432 1 482 3 554 30 2 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *4bcx6kbbrg6bv 3781789023 2248191382 0 11 0 41 2 26wh3untj05apd 3457450300 3233890669 0 11 0 131 1 206wh3untj05apd 3477405755 3233890669 0 11 1 131 2 208pzsjt5p64xfu 3998876049 3667423051 0 11 5 44 3 18bpfzx2hxf5x7f 1890295626 774548604 0 11 0 26 1 24g67nkxd2nqqqd 1308088852 4202046543 0 11 1 57 1 49g67nkxd2nqqqd 1308088852 1991738870 0 11 1 39 1 38g67nkxd2nqqqd 2154937993 1991738870 1 11 27 39 72 38g67nkxd2nqqqd 2154937993 4202046543 1 11 27 57 72 4992 rows selected.Elapsed: 00:00:02.53SQL>36 © 2009/2010 Pythian
  • SQL Plan flipping Example (3) … • In the result … • Load on the system was reduced by 5 times • Takeaways … A. SQL Plans may flip from good plans to … B. SQL Outlines/Profiles may help some times C. AWR provides good input for such analysis • Why SQL Plans may flip? 1. Bind variable peeking / adaptive cursor sharing 2. Statistics change (including difference in partitions stats) 3. Adding/Removing indexes 4. Session/System init.ora parameters (nls_sort/optimizer_mode) 5. Dynamic statistics gathering (sampling) 6. Profiles/Outlines/Baselines evolution37 © 2009/2010 Pythian
  • Conclusions … • AWR = DBA_HIST% views ( snapshots from V$% views ) • Sometimes it is the only source of information • AWR contains much more information that default AWR reports and Grid Control could provide you • Be careful mining data (there are some gotchas) • Don’t be afraid to discover/mine the AWR data I can show you the door … … but it is you who should walk through it38 © 2009/2010 Pythian
  • Pythian Facts• Founded in 1997, over 14 years• 100+ DBAs ! (140 employees)• 5 offices in 5 countries (true follow the sun model)• Employ • 6 Oracle ACEs (Including 1 ACE director) • Several Oracle Masters • Plenty of technical geeks• Platinum level partner in the Oracle Partner Network• Actively supports technical communities via • Blogging • Conferences • SIGs and other events39 © 2009/2010 Pythian
  • Google Yury Oracle [LinkedIn, twitter, blog, email, mobile, …]Additionalget the presentation Email me to Resources • www.oracle.com/scan • www.pythian.com/exadata • www.pythian.com/news/tag/exadata - Exadata Mission Blog Let you remember/consider AWR • www.pythian.com/news_and_events/in_the_newsArticle: “Making the Most time youExadata” next of Oracle troubleshoot Performance issue!My Oracle Support notes 888828.1 and 757552.1Thank you! Pythian •Like Pythian on facebook: http://on.fb.me/pythianfacebook •Follow us on LinkedIn: http://linkd.in/pythian •Follow Pythian on Twitter @pythian (www.twitter.com/pythian)40 © 2009/2010 Pythian