AWR Data mining


Yury Velikanov Senior Oracle DBA
Why Companies Trust Pythian
    • Recognized Leader:
    •   Global industry-leader in remote database administration services and consulting for Oracle,
        Oracle Applications, MySQL and SQL Server

    •   Work with over 150 multinational companies such as Forbes.com, Fox Sports, Nordion and Western
        Union to help manage their complex IT deployments

    • Expertise:
    •   One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 6 Oracle
        ACEs/ACE Directors.
    •   Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle
        GoldenGate & Oracle RAC.

    • Global Reach & Scalability:
    •   24/7/365 global remote support for DBA and consulting, systems administration, special projects
        or emergency response




2                                                  © 2011 Pythian
Mission




          Let you remember/consider AWR
             next time you troubleshoot
                 Performance issue!




3                   © 2009/2010 Pythian
NOTE




        AWR = STATSPACK              = Performance Repository




                                               Excerpt from
                       11GR2:$OH/rdbms/admin/awrrpt.sql
       “Rem   This report is based on the Statspack report.”
4              © 2009/2010 Pythian
AWR Agenda

    • Introduction & Background

    • Examples, Examples, Examples

    • Concept & Approach

    • More examples

    • Q&A
                                                         Google: Oracle Yury
                           Blog, Twitter, Linkedin, ACE … email, phone number

5                      © 2009/2010 Pythian
Few words about Yury
    • Google Yury Oracle [LinkedIn, twitter, blog, email, mobile, …]
            -   Email me to get the presentation

    • Sr. Oracle DBA at Pythian, Oracle ACE and OCM
    • Started as Oracle DBA
            -   with 7.2 (in 1997, 14+)
    • First international appearance
            -   2005 - Hotsos Symposium 2005
    • Professional Education
            -   Jonathan Lewis (2003 – 3 days), Tom Kyte (2004 – 3 days), Tanel
                Põder (2008 – 2 days ), Cary Millsap (2011 – 3 days) …
    • Education (Master Degree in Computer science)
            -   OCP 7/8/8i/9/10/11 + OCM 9i/10g/11g
    • Oracle DBA consultant experience (14+ years)
    • Pythian Oracle Clients support (2+ years)
            -   140+ Clients around the world
            -   Different products, different load, different problems
6                                     © 2009/2010 Pythian
Background
    • AWR is one of many RDBMS performance data sources
    Jonathan Lewis / Tom Kyte / Tanel Põder / Cary Millsap
    • Sometimes it isn’t the best source (aggregation)
         • SQL Extended trace (event 10046)
             • RAW trace
             • tkprof
             • TRCAnlzr [ID 224270.1]
             • Method-R state of art tools
         • PL/SQL Profiler
         • LTOM (Session Trace Collector)
         • others

    • Sometimes it is the best source!
    • Sometimes it is the only one available!

7                               © 2009/2010 Pythian
Background
    • Once I was called to troubleshoot high load
         • Connected to the database I saw 8 active processes
           running for 6 hours in average at the time I connected
         • I switched 10046 event for all 8 collected 15 minutes
           data and analyzed it one by one
         • Found several SQLs returning 1 row million times
         • Passed the results to development asking to fix the logic
         • Spent ~2 hours to find where the issue was

    • Next day a workmate asked me
         • Why did you use 10046 and spent 2 hours?
         • He used AWR report and came up with the same
           answer in less than 5 minutes

    • Lesson learned: Right tool for the right (job - no) case !
8                               © 2009/2010 Pythian
When should you consider AWR mining?
    • General resource tuning (high CPU, IO utilization)
         • You are asked to reduce server load X times

    • You would like to analyze load patterns/trends

    • You need to go back in time and see how things progressed

    • You don’t have any other source of information

    • Existing official AWR interface doesn’t provide you
      information at the right angle/dimension or are not available
      (Grid Control, awrrpt.sql)

    • AWR SQL Execution Plans historical information analysis

9                              © 2009/2010 Pythian
TOP CPU/IO Consuming SQLs ?
     select
              s.SQL_ID,
              sum(CPU_TIME_DELTA),
              sum(DISK_READS_DELTA),
              count(*)
     from
             DBA_HIST_SQLSTAT
     group by
             SQL_ID
     order by
             sum(CPU_TIME_DELTA) desc
     /

     SQL_ID        SUM(CPU_TIME_DELTA) SUM(DISK_READS_DELTA)   COUNT(*)
     ------------- ------------------- --------------------- ----------
     05s9358mm6vrr            27687500                  2940          1
     f6cz4n8y72xdc             7828125                  4695          2
     5dfmd823r8dsp             6421875                     8         15
     3h1rjtcff3wy1             5640625                   113          1
     92mb1kvurwn8h             5296875                     0          1
     bunssq950snhf             3937500                    18         15
     7xa8wfych4mad             2859375                     0          2
     ...




10                                      © 2009/2010 Pythian
TOP CPU Consuming SQLs ?

     select
              s.SQL_ID,
              sum(s.CPU_TIME_DELTA),
              sum(s.DISK_READS_DELTA),
              count(*)
     from
              DBA_HIST_SQLSTAT s




     group by
             s.SQL_ID
     order by
             sum(s.CPU_TIME_DELTA) desc




11                                       © 2009/2010 Pythian
TOP CPU Consuming SQLs ?
     select * from
     (
     select
             s.SQL_ID,
             sum(s.CPU_TIME_DELTA),
             sum(s.DISK_READS_DELTA),
             count(*)
     from
             DBA_HIST_SQLSTAT s




     group by
             s.SQL_ID
     order by
             sum(s.CPU_TIME_DELTA) desc
     )
     where rownum < 11
     /




12                                      © 2009/2010 Pythian
TOP CPU Consuming SQLs ?
     select * from
     (
     select
             s.SQL_ID,
             sum(s.CPU_TIME_DELTA),
             sum(s.DISK_READS_DELTA),
             count(*)
     from
             DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p
     where 1=1
             and s.SNAP_ID = p.SNAP_ID

            and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16


     group by
             s.SQL_ID
     order by
             sum(s.CPU_TIME_DELTA) desc
     )
     where rownum < 11
     /




13                                   © 2009/2010 Pythian
TOP CPU Consuming SQLs ?
     select * from
     (
     select
             s.SQL_ID,
             sum(s.CPU_TIME_DELTA),
             sum(s.DISK_READS_DELTA),
             count(*)
     from
             DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p
     where 1=1
             and s.SNAP_ID = p.SNAP_ID

            and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16

             and p.END_INTERVAL_TIME between SYSDATE-7 and SYSDATE
     group by
             s.SQL_ID
     order by
             sum(s.CPU_TIME_DELTA) desc
     )
     where rownum < 11
     /




14                                   © 2009/2010 Pythian
TOP CPU Consuming SQLs ?
     select * from
     (
     select
             s.SQL_ID,
             sum(s.CPU_TIME_DELTA),
             sum(s.DISK_READS_DELTA),
             count(*)
     from
             DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p, DBA_HIST_SQLTEXT t
     where 1=1
             and s.SNAP_ID = p.SNAP_ID
             and s.SQL_ID = t.SQL_ID
             and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16
             and t.COMMAND_TYPE != 47 –- Exclude PL/SQL blocks from output
             and p.END_INTERVAL_TIME between SYSDATE-7 and SYSDATE
     group by
             s.SQL_ID
     order by
             sum(s.CPU_TIME_DELTA) desc
     )
     where rownum < 11
     /




15                                   © 2009/2010 Pythian
TOP CPU Consuming SQLs ?


               2.                         3.
     5.                                                 1.




                                               52.8 %
                                                  4.




16                  © 2009/2010 Pythian
TOP CPU Consuming SQLs ?
     select
              SQL_ID,
              sum(CPU_TIME_DELTA),
              sum(DISK_READS_DELTA),
              count(*)
     from
             DBA_HIST_SQLSTAT
     group by
             SQL_ID
     order by
             sum(CPU_TIME_DELTA) desc
     /

     SQL_ID        SUM(CPU_TIME_DELTA) SUM(DISK_READS_DELTA)   COUNT(*)
     ------------- ------------------- --------------------- ----------
     05s9358mm6vrr            27687500                  2940          1
     f6cz4n8y72xdc             7828125                  4695          2
     5dfmd823r8dsp             6421875                     8         15
     3h1rjtcff3wy1             5640625                   113          1
     92mb1kvurwn8h             5296875                     0          1
     bunssq950snhf             3937500                    18         15
     7xa8wfych4mad             2859375                     0          2
     ...




17                                      © 2009/2010 Pythian
5 Slides of
     Concept & Approach




18        © 2009/2010 Pythian
AWR = DBA_HIST_% Views +
     • 111 Views in 11.2.0.2.0

     • I use just few on a regular basis
           •   DBA_HIST_ACTIVE_SESS_HISTORY                  - V$ACTIVE_SESSION_HISTORY
           •   DBA_HIST_SEG_STAT                             - V$SEGMENT_STATISTICS
           •   DBA_HIST_SQLSTAT                              - V$SQL
           •   DBA_HIST_SQL_PLAN                             - V$SQL_PLAN
           •   DBA_HIST_SYSSTAT                              - V$SYSSTAT ( ~SES~ )
           •   DBA_HIST_SYSTEM_EVENT                         - V$SYSTEM_EVENT ( ~SESSION~ )


     • Most of the views contain data snapshots from V$___ views

     • DELTA columns (e.g. DISK_READS_DELTA)
           • DBA_HIST_SEG_STAT
           • DBA_HIST_SQLSTAT




19                                     © 2009/2010 Pythian
AWR Things to keep in mind …
     • The data are just snapshots of V$ views

           • Data collected based on thresholds
              • Some data is excluded based on thresholds

           • Some data may not be in SGA at the time of snapshot
               • Longer time difference between snapshots more
                 data got excluded
               • For data mining use ALL snapshots available

                                                      Begin


                                                              End
                                                t
20                              © 2009/2010 Pythian
AWR Things to keep in mind …
     • Forget about AWR if there are constants in the code
          • Indicator is high parse count (hard) (10-50 per/sec)
          • It isn’t just hard parsing! (related bugs)
          • cursor_sharing = FORCE

     • In RAC configuration do not forget INST_ID column in joins

     • Most of the V$ (DBA_HIST) performance views have
       incremental counters. END - BEGIN values
           • You may get wrong results (sometimes negative)
           • Sometimes counters reach max value and get reset
           • Counters got reset at instance restart time

     • Time between snapshots may be different
          • Suggestion (ENDv - BEGINv)/(ENDs - BEGINs)=value/sec
21                                © 2009/2010 Pythian
22
                      -800000
                                -600000
                                          -400000
                                                    -200000
                                                                  200000
                                                                           400000
                                                                                    600000
                                                                                             800000




                                                              0




© 2009/2010 Pythian
                                                                                                                150
                                                                                                                200
                                                                                                                250


                                                                                                                100
                                                                                                                 50
                                                                                                                  0




                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
                                                                                                                      AWR Things to keep in mind …




                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
                                                                                                      2011.10.22…
AWR Things to keep in mind …
     • Seconds count between 2 timestamps
     select
       s.BEGIN_INTERVAL_TIME,
       s.END_INTERVAL_TIME,
       s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME DTIME, -- Returns “Interval”

       EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) H,
       EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) M,
       EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) S,

       EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+
       EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+
       EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) SECS,

       phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) -– Write you own fun()
       (cast(s.END_INTERVAL_TIME as date) - cast(s.BEGIN_INTERVAL_TIME as date))
        *24*60*60
     from
       DBA_HIST_SNAPSHOT s
     where 1=1
       and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE)
       and s.DBID = (select DBID from V$DATABASE)
     order by
       s.BEGIN_INTERVAL_TIME;

23                                    © 2009/2010 Pythian
AWR Things to keep in mind …
     select SNAP_INTERVAL, RETENTION
     from
             DBA_HIST_WR_CONTROL c, V$DATABASE d
     where
             c.DBID = d.DBID;

     SNAP_INTERVAL                  RETENTION
     ------------------------------ ------------------------------
     +00000 01:00:00.0              +00007 00:00:00.0


     select DBID, INSTANCE_NUMBER, count(*) C,
       min(BEGIN_INTERVAL_TIME) OLDEST, max(BEGIN_INTERVAL_TIME) YUNGEST
     from
       DBA_HIST_SNAPSHOT
     group by
       DBID,
       INSTANCE_NUMBER;

      DBID INSTANCE_NUMBER          C OLDEST                    YUNGEST
     ---------- --------------- ---------- ------------------------- -------------------------
     3244685755               1        179 13-AUG-11 07.00.30.233 PM 21-AUG-11 05.00.01.855 AM
     3244685755               2        179 13-AUG-11 07.00.30.309 PM 21-AUG-11 05.00.01.761 AM




24                                        © 2009/2010 Pythian
Trends Analysis Example (1) …
                  DBA_HIST_SYSSTAT & DBA_HIST_SYSTEM_EVENT

     select
       s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME,

      (
       t.VALUE-
               LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME)
       ) DVALUE,

       (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/
       phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) VAL_SEC
     from
       DBA_HIST_SNAPSHOT s,
       DBA_HIST_SYSSTAT t
     where 1=1
       and s.SNAP_ID = t.SNAP_ID
       and s.DBID = t.DBID
       and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER
       and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE)
       and s.DBID = (select DBID from V$DATABASE)
       and t.STAT_NAME = 'parse count (hard)'
     order by
       s.BEGIN_INTERVAL_TIME;

25                                   © 2009/2010 Pythian
Trends Analysis Example (1) …




26                 © 2009/2010 Pythian
Trends Analysis Example (1) …
                  DBA_HIST_SYSSTAT & DBA_HIST_SYSTEM_EVENT

     select
       s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME,

      (
       t.VALUE-
               LAG (t.VALUE) OVER (ORDER BY s.END_INTERVAL_TIME)
       ) DVALUE,

       (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.END_INTERVAL_TIME))/
       phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) VAL_SEC
     from
       DBA_HIST_SNAPSHOT s,
       DBA_HIST_SYSSTAT t
     where 1=1
       and s.SNAP_ID = t.SNAP_ID
       and s.DBID = t.DBID
       and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER
       and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE)
       and s.DBID = (select DBID from V$DATABASE)
       and t.STAT_NAME = 'parse count (hard)'
     order by
       s.END_INTERVAL_TIME;

27                                   © 2009/2010 Pythian
SQL Bad performance Example (2) …
     • Called by a client to troubleshoot a SQL with bad performance

     • Sometimes the SQL hangs (never finishes) and needs to be killed
       and re-executed

     • Upon re-execution, it always finishes successfully in a few
       minutes

     • The client demanded a resolution ASAP …




28                              © 2009/2010 Pythian
SQL Bad performance Example (2) …
                                  DBA_HIST_SQLSTAT


     select
       st.SQL_ID
     , st.PLAN_HASH_VALUE
     , sum(st.EXECUTIONS_DELTA) EXECUTIONS
     , sum(st.ROWS_PROCESSED_DELTA) CROWS
     , trunc(sum(st.CPU_TIME_DELTA)/1000000/60) CPU_MINS
     , trunc(sum(st.ELAPSED_TIME_DELTA)/1000000/60) ELA_MINS
     from DBA_HIST_SQLSTAT st
     where st.SQL_ID in (
     '5ppdcygtcw7p6'
     ,'gpj32cqd0qy9a'
     )
     group by st.SQL_ID , st.PLAN_HASH_VALUE
     order by st.SQL_ID, CPU_MINS;




29                                  © 2009/2010 Pythian
SQL Bad performance Example (2) …
                                       DBA_HIST_SQLSTAT


     SQL_ID        PLAN_HASH_VALUE EXECUTIONS      CROWS         CPU_MINS         ELA_MINS
     ------------- --------------- ---------- ---------- ---------------- ----------------
     5ppdcygtcw7p6       436796090         20      82733                1                3
     5ppdcygtcw7p6       863350916         71     478268                5               11
     5ppdcygtcw7p6      2817686509          9      32278            2,557            2,765


     gpj32cqd0qy9a     3094138997          30         58400             1                3
     gpj32cqd0qy9a     1700210966          36         69973             1                7
     gpj32cqd0qy9a     1168845432           2           441           482              554
     gpj32cqd0qy9a     2667660534           4          1489         1,501            1,642




30                                        © 2009/2010 Pythian
SQL Bad performance Example (2) …
                    DBA_HIST_SQLSTAT & DBA_HIST_SEG_STAT


     select
       st.SQL_ID
     , st.PLAN_HASH_VALUE
     , sum(st.EXECUTIONS_DELTA) EXECUTIONS
     , sum(st.ROWS_PROCESSED_DELTA) CROWS
     , trunc(sum(st.CPU_TIME_DELTA)/1000000/60) CPU_MINS
     , trunc(sum(st.ELAPSED_TIME_DELTA)/1000000/60) ELA_MINS
     from DBA_HIST_SQLSTAT st
     where st.SQL_ID in (
     '5ppdcygtcw7p6'
     ,'gpj32cqd0qy9a'
     )
     group by st.SQL_ID , st.PLAN_HASH_VALUE
     order by st.SQL_ID, CPU_MINS;




31                                  © 2009/2010 Pythian
SQL Bad performance Example (2) …
     • In the result …

     • Two different jobs were gathering statistics on a daily basis
         1. “ANALYZE …” part of other batch job (developer)
         2. “DBMS_STATS…” traditional (DBA)

     • Sometimes “DBMS_STATS…“ is not completed before the batch
       job starts (+/- 10 minutes).

     • After the job got killed (typically well after 10 mins since start)
       the new “correct” statistics were in place.

     • Takeaways …
          A. Don’t change your statistics that frequently (should be consistent)
          B. AWR data helps to spot such issues easily
32                                  © 2009/2010 Pythian
SQL Plan flipping Example (3) …
     • I asked myself: Well !

           • If you find that the execution plan for a SQL has changed
             from a good (quick) to a bad one (slow), how do you know
             if there are other affected SQLs?

           • And if there are, how many and which ones?

           • Would SQL Profiles (Outlines) help address those?




33                              © 2009/2010 Pythian
SQL Plan flipping Example (3) …
     SELECT st2.SQL_ID ,
       st2.PLAN_HASH_VALUE ,
       st_long.PLAN_HASH_VALUE l_PLAN_HASH_VALUE ,
       st2.CPU_MINS ,
       st_long.CPU_MINS l_CPU_MINS ,
       st2.ELA_MINS ,
       st_long.ELA_MINS l_ELA_MINS ,
       st2.EXECUTIONS ,
       st_long.EXECUTIONS l_EXECUTIONS ,
       st2.CROWS ,
       st_long.CROWS l_CROWS ,
       st2.CPU_MINS_PER_ROW ,
       st_long.CPU_MINS_PER_ROW l_CPU_MINS_PER_ROW
     FROM
       (SELECT st.SQL_ID ,
          st.PLAN_HASH_VALUE ,
          SUM(st.EXECUTIONS_DELTA) EXECUTIONS ,
          SUM(st.ROWS_PROCESSED_DELTA) CROWS ,
          TRUNC(SUM(st.CPU_TIME_DELTA)                                         /1000000/60) CPU_MINS ,
          DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW ,
          TRUNC(SUM(st.ELAPSED_TIME_DELTA)                                     /1000000/60) ELA_MINS
       FROM DBA_HIST_SQLSTAT st
       WHERE 1                      =1
       AND ( st.CPU_TIME_DELTA     !=0
       OR st.ROWS_PROCESSED_DELTA !=0)
       GROUP BY st.SQL_ID,
          st.PLAN_HASH_VALUE
       ) st2,
       (SELECT st.SQL_ID ,
          st.PLAN_HASH_VALUE ,
          SUM(st.EXECUTIONS_DELTA) EXECUTIONS ,
          SUM(st.ROWS_PROCESSED_DELTA) CROWS ,
          TRUNC(SUM(st.CPU_TIME_DELTA)                                         /1000000/60) CPU_MINS ,
          DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW ,
          TRUNC(SUM(st.ELAPSED_TIME_DELTA)                                     /1000000/60) ELA_MINS
       FROM DBA_HIST_SQLSTAT st
       WHERE 1                                          =1
       AND ( st.CPU_TIME_DELTA                         !=0
       OR st.ROWS_PROCESSED_DELTA                      !=0)
       HAVING TRUNC(SUM(st.CPU_TIME_DELTA)/1000000/60) > 10
       GROUP BY st.SQL_ID,
          st.PLAN_HASH_VALUE
       ) st_long
     WHERE 1                                                                             =1
     AND st2.SQL_ID                                                                      = st_long.SQL_ID
     AND st_long.CPU_MINS_PER_ROW/DECODE(st2.CPU_MINS_PER_ROW,0,1,st2.CPU_MINS_PER_ROW) > 2
     ORDER BY l_CPU_MINS DESC,
       st2.SQL_ID,
       st_long.CPU_MINS DESC,
       st2.PLAN_HASH_VALUE;




34                                                                   © 2009/2010 Pythian
SQL Plan flipping Example (3) …
     SELECT
       ...
     FROM
       (SELECT st.SQL_ID ,
          st.PLAN_HASH_VALUE ,
          ...
          DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0
     , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW ,
          ...
       FROM DBA_HIST_SQLSTAT st
       WHERE 1                      =1
       ...
       GROUP BY st.SQL_ID,
          st.PLAN_HASH_VALUE
       ) st2,
       (SELECT st.SQL_ID ,
          st.PLAN_HASH_VALUE ,
          ...
     HAVING trunc(sum(st.CPU_TIME_DELTA)/1000000/60) > 10
     GROUP BY st.SQL_ID,
          st.PLAN_HASH_VALUE
       ) st_long
     WHERE 1                                                                            =1
     AND st2.SQL_ID                                                                     =
     st_long.SQL_ID
     AND st_long.CPU_MINS_PER_ROW/DECODE(st2.CPU_MINS_PER_ROW,0,1,st2.CPU_MINS_PER_ROW) > 2
     ORDER BY l_CPU_MINS DESC,
       st2.SQL_ID,
       st_long.CPU_MINS DESC,
       st2.PLAN_HASH_VALUE;


35                                        © 2009/2010 Pythian
SQL Plan flipping Example (3) …
SQL_ID        PLAN_HASH_VALUE L_PLAN_HASH_VALUE   CPU_MINS L_CPU_MINS   ELA_MINS L_ELA_MINS EXECUTIONS L_EXECUTIONS
------------- --------------- ----------------- ---------- ---------- ---------- ---------- ---------- ------------
db8yz0rfhvufm      3387634876         619162475         17       2673         21       4074    3106638          193
5ppdcygtcw7p6       436796090        2817686509          1       2557          3       2765         20            9
5ppdcygtcw7p6       863350916        2817686509          5       2557         11       2765         71            9
1tab7mjut8j9h       875484785         911605088          9       2112         23       2284        980         1436
1tab7mjut8j9h      2484900321         911605088          6       2112          6       2284       1912         1436
1tab7mjut8j9h      3141038411         911605088         50       2112         57       2284      32117         1436
gpj32cqd0qy9a      1700210966        2667660534          1       1501          7       1642         36            4
gpj32cqd0qy9a      3094138997        2667660534          1       1501          3       1642         30            4
2tf4p2anpwpk2       825403357        1679851684          6        824         71        913         17           13
csvwu3kqu43j4      3860135778        2851322291          0        784          0        874          1            2
0q9hpmtk8c1hf      3860135778        2851322291          0        779          0        867          1            2
2frwhbxvg1j69      3860135778        2851322291          0        776          0        865          1            2
4nzsxm3d9rspt      3860135778        2851322291          0        754          0        846          1            2
1pc2npdb1kbp6         9772089        2800812079          0        511          0       3000          7          695
gpj32cqd0qy9a      1700210966        1168845432          1        482          7        554         36            2
gpj32cqd0qy9a      3094138997        1168845432          1        482          3        554         30            2
 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
4bcx6kbbrg6bv      3781789023        2248191382          0         11          0         41          2            2
6wh3untj05apd      3457450300        3233890669          0         11          0        131          1           20
6wh3untj05apd      3477405755        3233890669          0         11          1        131          2           20
8pzsjt5p64xfu      3998876049        3667423051          0         11          5         44          3           18
bpfzx2hxf5x7f      1890295626         774548604          0         11          0         26          1           24
g67nkxd2nqqqd      1308088852        4202046543          0         11          1         57          1           49
g67nkxd2nqqqd      1308088852        1991738870          0         11          1         39          1           38
g67nkxd2nqqqd      2154937993        1991738870          1         11         27         39         72           38
g67nkxd2nqqqd      2154937993        4202046543          1         11         27         57         72           49

92 rows selected.

Elapsed: 00:00:02.53
SQL>




36                                              © 2009/2010 Pythian
SQL Plan flipping Example (3) …
     • In the result …

     • Load on the system was reduced by 5 times
     • Takeaways …
          A. SQL Plans may flip from good plans to …
          B. SQL Outlines/Profiles may help some times
          C. AWR provides good input for such analysis

     • Why SQL Plans may flip?
         1. Bind variable peeking / adaptive cursor sharing
         2. Statistics change (including difference in partitions stats)
         3. Adding/Removing indexes
         4. Session/System init.ora parameters (nls_sort/optimizer_mode)
         5. Dynamic statistics gathering (sampling)
         6. Profiles/Outlines/Baselines evolution
37                               © 2009/2010 Pythian
Conclusions …
     • AWR = DBA_HIST% views ( snapshots from V$% views )

     • Sometimes it is the only source of information

     • AWR contains much more information that default AWR reports
       and Grid Control could provide you

     • Be careful mining data (there are some gotchas)

     • Don’t be afraid to discover/mine the AWR data


     I can show you the door …
                          … but it is you who should walk through it

38                              © 2009/2010 Pythian
Pythian Facts
•     Founded in 1997, over 14 years
•     100+ DBAs ! (140 employees)
•     5 offices in 5 countries (true follow the sun model)
•     Employ
         •   6 Oracle ACEs (Including 1 ACE director)
         •   Several Oracle Masters
         •   Plenty of technical geeks
• Platinum level partner in the Oracle Partner Network
• Actively supports technical communities via
     • Blogging
     • Conferences
     • SIGs and other events



39                              © 2009/2010 Pythian
Google Yury Oracle [LinkedIn, twitter, blog, email, mobile, …]
Additionalget the presentation
  Email me to Resources

 • www.oracle.com/scan
 • www.pythian.com/exadata
 • www.pythian.com/news/tag/exadata - Exadata
         Mission
   Blog
                Let you remember/consider AWR
 • www.pythian.com/news_and_events/in_the_news
Article: “Making the Most time youExadata”
                    next of Oracle troubleshoot
                           Performance issue!
My Oracle Support notes 888828.1 and 757552.1


Thank you!
                          Pythian
                             •Like Pythian on facebook: http://on.fb.me/pythianfacebook
                             •Follow us on LinkedIn: http://linkd.in/pythian
                             •Follow Pythian on Twitter @pythian (www.twitter.com/pythian)

40                           © 2009/2010 Pythian

Oracle AWR Data mining

  • 1.
    AWR Data mining YuryVelikanov Senior Oracle DBA
  • 2.
    Why Companies TrustPythian • Recognized Leader: • Global industry-leader in remote database administration services and consulting for Oracle, Oracle Applications, MySQL and SQL Server • Work with over 150 multinational companies such as Forbes.com, Fox Sports, Nordion and Western Union to help manage their complex IT deployments • Expertise: • One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 6 Oracle ACEs/ACE Directors. • Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle GoldenGate & Oracle RAC. • Global Reach & Scalability: • 24/7/365 global remote support for DBA and consulting, systems administration, special projects or emergency response 2 © 2011 Pythian
  • 3.
    Mission Let you remember/consider AWR next time you troubleshoot Performance issue! 3 © 2009/2010 Pythian
  • 4.
    NOTE AWR = STATSPACK = Performance Repository Excerpt from 11GR2:$OH/rdbms/admin/awrrpt.sql “Rem This report is based on the Statspack report.” 4 © 2009/2010 Pythian
  • 5.
    AWR Agenda • Introduction & Background • Examples, Examples, Examples • Concept & Approach • More examples • Q&A Google: Oracle Yury Blog, Twitter, Linkedin, ACE … email, phone number 5 © 2009/2010 Pythian
  • 6.
    Few words aboutYury • Google Yury Oracle [LinkedIn, twitter, blog, email, mobile, …] - Email me to get the presentation • Sr. Oracle DBA at Pythian, Oracle ACE and OCM • Started as Oracle DBA - with 7.2 (in 1997, 14+) • First international appearance - 2005 - Hotsos Symposium 2005 • Professional Education - Jonathan Lewis (2003 – 3 days), Tom Kyte (2004 – 3 days), Tanel Põder (2008 – 2 days ), Cary Millsap (2011 – 3 days) … • Education (Master Degree in Computer science) - OCP 7/8/8i/9/10/11 + OCM 9i/10g/11g • Oracle DBA consultant experience (14+ years) • Pythian Oracle Clients support (2+ years) - 140+ Clients around the world - Different products, different load, different problems 6 © 2009/2010 Pythian
  • 7.
    Background • AWR is one of many RDBMS performance data sources Jonathan Lewis / Tom Kyte / Tanel Põder / Cary Millsap • Sometimes it isn’t the best source (aggregation) • SQL Extended trace (event 10046) • RAW trace • tkprof • TRCAnlzr [ID 224270.1] • Method-R state of art tools • PL/SQL Profiler • LTOM (Session Trace Collector) • others • Sometimes it is the best source! • Sometimes it is the only one available! 7 © 2009/2010 Pythian
  • 8.
    Background • Once I was called to troubleshoot high load • Connected to the database I saw 8 active processes running for 6 hours in average at the time I connected • I switched 10046 event for all 8 collected 15 minutes data and analyzed it one by one • Found several SQLs returning 1 row million times • Passed the results to development asking to fix the logic • Spent ~2 hours to find where the issue was • Next day a workmate asked me • Why did you use 10046 and spent 2 hours? • He used AWR report and came up with the same answer in less than 5 minutes • Lesson learned: Right tool for the right (job - no) case ! 8 © 2009/2010 Pythian
  • 9.
    When should youconsider AWR mining? • General resource tuning (high CPU, IO utilization) • You are asked to reduce server load X times • You would like to analyze load patterns/trends • You need to go back in time and see how things progressed • You don’t have any other source of information • Existing official AWR interface doesn’t provide you information at the right angle/dimension or are not available (Grid Control, awrrpt.sql) • AWR SQL Execution Plans historical information analysis 9 © 2009/2010 Pythian
  • 10.
    TOP CPU/IO ConsumingSQLs ? select s.SQL_ID, sum(CPU_TIME_DELTA), sum(DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT group by SQL_ID order by sum(CPU_TIME_DELTA) desc / SQL_ID SUM(CPU_TIME_DELTA) SUM(DISK_READS_DELTA) COUNT(*) ------------- ------------------- --------------------- ---------- 05s9358mm6vrr 27687500 2940 1 f6cz4n8y72xdc 7828125 4695 2 5dfmd823r8dsp 6421875 8 15 3h1rjtcff3wy1 5640625 113 1 92mb1kvurwn8h 5296875 0 1 bunssq950snhf 3937500 18 15 7xa8wfych4mad 2859375 0 2 ... 10 © 2009/2010 Pythian
  • 11.
    TOP CPU ConsumingSQLs ? select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc 11 © 2009/2010 Pythian
  • 12.
    TOP CPU ConsumingSQLs ? select * from ( select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc ) where rownum < 11 / 12 © 2009/2010 Pythian
  • 13.
    TOP CPU ConsumingSQLs ? select * from ( select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p where 1=1 and s.SNAP_ID = p.SNAP_ID and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16 group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc ) where rownum < 11 / 13 © 2009/2010 Pythian
  • 14.
    TOP CPU ConsumingSQLs ? select * from ( select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p where 1=1 and s.SNAP_ID = p.SNAP_ID and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16 and p.END_INTERVAL_TIME between SYSDATE-7 and SYSDATE group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc ) where rownum < 11 / 14 © 2009/2010 Pythian
  • 15.
    TOP CPU ConsumingSQLs ? select * from ( select s.SQL_ID, sum(s.CPU_TIME_DELTA), sum(s.DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p, DBA_HIST_SQLTEXT t where 1=1 and s.SNAP_ID = p.SNAP_ID and s.SQL_ID = t.SQL_ID and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16 and t.COMMAND_TYPE != 47 –- Exclude PL/SQL blocks from output and p.END_INTERVAL_TIME between SYSDATE-7 and SYSDATE group by s.SQL_ID order by sum(s.CPU_TIME_DELTA) desc ) where rownum < 11 / 15 © 2009/2010 Pythian
  • 16.
    TOP CPU ConsumingSQLs ? 2. 3. 5. 1. 52.8 % 4. 16 © 2009/2010 Pythian
  • 17.
    TOP CPU ConsumingSQLs ? select SQL_ID, sum(CPU_TIME_DELTA), sum(DISK_READS_DELTA), count(*) from DBA_HIST_SQLSTAT group by SQL_ID order by sum(CPU_TIME_DELTA) desc / SQL_ID SUM(CPU_TIME_DELTA) SUM(DISK_READS_DELTA) COUNT(*) ------------- ------------------- --------------------- ---------- 05s9358mm6vrr 27687500 2940 1 f6cz4n8y72xdc 7828125 4695 2 5dfmd823r8dsp 6421875 8 15 3h1rjtcff3wy1 5640625 113 1 92mb1kvurwn8h 5296875 0 1 bunssq950snhf 3937500 18 15 7xa8wfych4mad 2859375 0 2 ... 17 © 2009/2010 Pythian
  • 18.
    5 Slides of Concept & Approach 18 © 2009/2010 Pythian
  • 19.
    AWR = DBA_HIST_%Views + • 111 Views in 11.2.0.2.0 • I use just few on a regular basis • DBA_HIST_ACTIVE_SESS_HISTORY - V$ACTIVE_SESSION_HISTORY • DBA_HIST_SEG_STAT - V$SEGMENT_STATISTICS • DBA_HIST_SQLSTAT - V$SQL • DBA_HIST_SQL_PLAN - V$SQL_PLAN • DBA_HIST_SYSSTAT - V$SYSSTAT ( ~SES~ ) • DBA_HIST_SYSTEM_EVENT - V$SYSTEM_EVENT ( ~SESSION~ ) • Most of the views contain data snapshots from V$___ views • DELTA columns (e.g. DISK_READS_DELTA) • DBA_HIST_SEG_STAT • DBA_HIST_SQLSTAT 19 © 2009/2010 Pythian
  • 20.
    AWR Things tokeep in mind … • The data are just snapshots of V$ views • Data collected based on thresholds • Some data is excluded based on thresholds • Some data may not be in SGA at the time of snapshot • Longer time difference between snapshots more data got excluded • For data mining use ALL snapshots available Begin End t 20 © 2009/2010 Pythian
  • 21.
    AWR Things tokeep in mind … • Forget about AWR if there are constants in the code • Indicator is high parse count (hard) (10-50 per/sec) • It isn’t just hard parsing! (related bugs) • cursor_sharing = FORCE • In RAC configuration do not forget INST_ID column in joins • Most of the V$ (DBA_HIST) performance views have incremental counters. END - BEGIN values • You may get wrong results (sometimes negative) • Sometimes counters reach max value and get reset • Counters got reset at instance restart time • Time between snapshots may be different • Suggestion (ENDv - BEGINv)/(ENDs - BEGINs)=value/sec 21 © 2009/2010 Pythian
  • 22.
    22 -800000 -600000 -400000 -200000 200000 400000 600000 800000 0 © 2009/2010 Pythian 150 200 250 100 50 0 2011.10.22… 2011.10.22… AWR Things to keep in mind … 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22… 2011.10.22…
  • 23.
    AWR Things tokeep in mind … • Seconds count between 2 timestamps select s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME, s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME DTIME, -- Returns “Interval” EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) H, EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) M, EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) S, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+ EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+ EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) SECS, phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) -– Write you own fun() (cast(s.END_INTERVAL_TIME as date) - cast(s.BEGIN_INTERVAL_TIME as date)) *24*60*60 from DBA_HIST_SNAPSHOT s where 1=1 and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) order by s.BEGIN_INTERVAL_TIME; 23 © 2009/2010 Pythian
  • 24.
    AWR Things tokeep in mind … select SNAP_INTERVAL, RETENTION from DBA_HIST_WR_CONTROL c, V$DATABASE d where c.DBID = d.DBID; SNAP_INTERVAL RETENTION ------------------------------ ------------------------------ +00000 01:00:00.0 +00007 00:00:00.0 select DBID, INSTANCE_NUMBER, count(*) C, min(BEGIN_INTERVAL_TIME) OLDEST, max(BEGIN_INTERVAL_TIME) YUNGEST from DBA_HIST_SNAPSHOT group by DBID, INSTANCE_NUMBER; DBID INSTANCE_NUMBER C OLDEST YUNGEST ---------- --------------- ---------- ------------------------- ------------------------- 3244685755 1 179 13-AUG-11 07.00.30.233 PM 21-AUG-11 05.00.01.855 AM 3244685755 2 179 13-AUG-11 07.00.30.309 PM 21-AUG-11 05.00.01.761 AM 24 © 2009/2010 Pythian
  • 25.
    Trends Analysis Example(1) … DBA_HIST_SYSSTAT & DBA_HIST_SYSTEM_EVENT select s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME, ( t.VALUE- LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) ) DVALUE, (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/ phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) VAL_SEC from DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT t where 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = 'parse count (hard)' order by s.BEGIN_INTERVAL_TIME; 25 © 2009/2010 Pythian
  • 26.
    Trends Analysis Example(1) … 26 © 2009/2010 Pythian
  • 27.
    Trends Analysis Example(1) … DBA_HIST_SYSSTAT & DBA_HIST_SYSTEM_EVENT select s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME, ( t.VALUE- LAG (t.VALUE) OVER (ORDER BY s.END_INTERVAL_TIME) ) DVALUE, (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.END_INTERVAL_TIME))/ phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) VAL_SEC from DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT t where 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = 'parse count (hard)' order by s.END_INTERVAL_TIME; 27 © 2009/2010 Pythian
  • 28.
    SQL Bad performanceExample (2) … • Called by a client to troubleshoot a SQL with bad performance • Sometimes the SQL hangs (never finishes) and needs to be killed and re-executed • Upon re-execution, it always finishes successfully in a few minutes • The client demanded a resolution ASAP … 28 © 2009/2010 Pythian
  • 29.
    SQL Bad performanceExample (2) … DBA_HIST_SQLSTAT select st.SQL_ID , st.PLAN_HASH_VALUE , sum(st.EXECUTIONS_DELTA) EXECUTIONS , sum(st.ROWS_PROCESSED_DELTA) CROWS , trunc(sum(st.CPU_TIME_DELTA)/1000000/60) CPU_MINS , trunc(sum(st.ELAPSED_TIME_DELTA)/1000000/60) ELA_MINS from DBA_HIST_SQLSTAT st where st.SQL_ID in ( '5ppdcygtcw7p6' ,'gpj32cqd0qy9a' ) group by st.SQL_ID , st.PLAN_HASH_VALUE order by st.SQL_ID, CPU_MINS; 29 © 2009/2010 Pythian
  • 30.
    SQL Bad performanceExample (2) … DBA_HIST_SQLSTAT SQL_ID PLAN_HASH_VALUE EXECUTIONS CROWS CPU_MINS ELA_MINS ------------- --------------- ---------- ---------- ---------------- ---------------- 5ppdcygtcw7p6 436796090 20 82733 1 3 5ppdcygtcw7p6 863350916 71 478268 5 11 5ppdcygtcw7p6 2817686509 9 32278 2,557 2,765 gpj32cqd0qy9a 3094138997 30 58400 1 3 gpj32cqd0qy9a 1700210966 36 69973 1 7 gpj32cqd0qy9a 1168845432 2 441 482 554 gpj32cqd0qy9a 2667660534 4 1489 1,501 1,642 30 © 2009/2010 Pythian
  • 31.
    SQL Bad performanceExample (2) … DBA_HIST_SQLSTAT & DBA_HIST_SEG_STAT select st.SQL_ID , st.PLAN_HASH_VALUE , sum(st.EXECUTIONS_DELTA) EXECUTIONS , sum(st.ROWS_PROCESSED_DELTA) CROWS , trunc(sum(st.CPU_TIME_DELTA)/1000000/60) CPU_MINS , trunc(sum(st.ELAPSED_TIME_DELTA)/1000000/60) ELA_MINS from DBA_HIST_SQLSTAT st where st.SQL_ID in ( '5ppdcygtcw7p6' ,'gpj32cqd0qy9a' ) group by st.SQL_ID , st.PLAN_HASH_VALUE order by st.SQL_ID, CPU_MINS; 31 © 2009/2010 Pythian
  • 32.
    SQL Bad performanceExample (2) … • In the result … • Two different jobs were gathering statistics on a daily basis 1. “ANALYZE …” part of other batch job (developer) 2. “DBMS_STATS…” traditional (DBA) • Sometimes “DBMS_STATS…“ is not completed before the batch job starts (+/- 10 minutes). • After the job got killed (typically well after 10 mins since start) the new “correct” statistics were in place. • Takeaways … A. Don’t change your statistics that frequently (should be consistent) B. AWR data helps to spot such issues easily 32 © 2009/2010 Pythian
  • 33.
    SQL Plan flippingExample (3) … • I asked myself: Well ! • If you find that the execution plan for a SQL has changed from a good (quick) to a bad one (slow), how do you know if there are other affected SQLs? • And if there are, how many and which ones? • Would SQL Profiles (Outlines) help address those? 33 © 2009/2010 Pythian
  • 34.
    SQL Plan flippingExample (3) … SELECT st2.SQL_ID , st2.PLAN_HASH_VALUE , st_long.PLAN_HASH_VALUE l_PLAN_HASH_VALUE , st2.CPU_MINS , st_long.CPU_MINS l_CPU_MINS , st2.ELA_MINS , st_long.ELA_MINS l_ELA_MINS , st2.EXECUTIONS , st_long.EXECUTIONS l_EXECUTIONS , st2.CROWS , st_long.CROWS l_CROWS , st2.CPU_MINS_PER_ROW , st_long.CPU_MINS_PER_ROW l_CPU_MINS_PER_ROW FROM (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , SUM(st.EXECUTIONS_DELTA) EXECUTIONS , SUM(st.ROWS_PROCESSED_DELTA) CROWS , TRUNC(SUM(st.CPU_TIME_DELTA) /1000000/60) CPU_MINS , DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW , TRUNC(SUM(st.ELAPSED_TIME_DELTA) /1000000/60) ELA_MINS FROM DBA_HIST_SQLSTAT st WHERE 1 =1 AND ( st.CPU_TIME_DELTA !=0 OR st.ROWS_PROCESSED_DELTA !=0) GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st2, (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , SUM(st.EXECUTIONS_DELTA) EXECUTIONS , SUM(st.ROWS_PROCESSED_DELTA) CROWS , TRUNC(SUM(st.CPU_TIME_DELTA) /1000000/60) CPU_MINS , DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW , TRUNC(SUM(st.ELAPSED_TIME_DELTA) /1000000/60) ELA_MINS FROM DBA_HIST_SQLSTAT st WHERE 1 =1 AND ( st.CPU_TIME_DELTA !=0 OR st.ROWS_PROCESSED_DELTA !=0) HAVING TRUNC(SUM(st.CPU_TIME_DELTA)/1000000/60) > 10 GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st_long WHERE 1 =1 AND st2.SQL_ID = st_long.SQL_ID AND st_long.CPU_MINS_PER_ROW/DECODE(st2.CPU_MINS_PER_ROW,0,1,st2.CPU_MINS_PER_ROW) > 2 ORDER BY l_CPU_MINS DESC, st2.SQL_ID, st_long.CPU_MINS DESC, st2.PLAN_HASH_VALUE; 34 © 2009/2010 Pythian
  • 35.
    SQL Plan flippingExample (3) … SELECT ... FROM (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , ... DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW , ... FROM DBA_HIST_SQLSTAT st WHERE 1 =1 ... GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st2, (SELECT st.SQL_ID , st.PLAN_HASH_VALUE , ... HAVING trunc(sum(st.CPU_TIME_DELTA)/1000000/60) > 10 GROUP BY st.SQL_ID, st.PLAN_HASH_VALUE ) st_long WHERE 1 =1 AND st2.SQL_ID = st_long.SQL_ID AND st_long.CPU_MINS_PER_ROW/DECODE(st2.CPU_MINS_PER_ROW,0,1,st2.CPU_MINS_PER_ROW) > 2 ORDER BY l_CPU_MINS DESC, st2.SQL_ID, st_long.CPU_MINS DESC, st2.PLAN_HASH_VALUE; 35 © 2009/2010 Pythian
  • 36.
    SQL Plan flippingExample (3) … SQL_ID PLAN_HASH_VALUE L_PLAN_HASH_VALUE CPU_MINS L_CPU_MINS ELA_MINS L_ELA_MINS EXECUTIONS L_EXECUTIONS ------------- --------------- ----------------- ---------- ---------- ---------- ---------- ---------- ------------ db8yz0rfhvufm 3387634876 619162475 17 2673 21 4074 3106638 193 5ppdcygtcw7p6 436796090 2817686509 1 2557 3 2765 20 9 5ppdcygtcw7p6 863350916 2817686509 5 2557 11 2765 71 9 1tab7mjut8j9h 875484785 911605088 9 2112 23 2284 980 1436 1tab7mjut8j9h 2484900321 911605088 6 2112 6 2284 1912 1436 1tab7mjut8j9h 3141038411 911605088 50 2112 57 2284 32117 1436 gpj32cqd0qy9a 1700210966 2667660534 1 1501 7 1642 36 4 gpj32cqd0qy9a 3094138997 2667660534 1 1501 3 1642 30 4 2tf4p2anpwpk2 825403357 1679851684 6 824 71 913 17 13 csvwu3kqu43j4 3860135778 2851322291 0 784 0 874 1 2 0q9hpmtk8c1hf 3860135778 2851322291 0 779 0 867 1 2 2frwhbxvg1j69 3860135778 2851322291 0 776 0 865 1 2 4nzsxm3d9rspt 3860135778 2851322291 0 754 0 846 1 2 1pc2npdb1kbp6 9772089 2800812079 0 511 0 3000 7 695 gpj32cqd0qy9a 1700210966 1168845432 1 482 7 554 36 2 gpj32cqd0qy9a 3094138997 1168845432 1 482 3 554 30 2 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 4bcx6kbbrg6bv 3781789023 2248191382 0 11 0 41 2 2 6wh3untj05apd 3457450300 3233890669 0 11 0 131 1 20 6wh3untj05apd 3477405755 3233890669 0 11 1 131 2 20 8pzsjt5p64xfu 3998876049 3667423051 0 11 5 44 3 18 bpfzx2hxf5x7f 1890295626 774548604 0 11 0 26 1 24 g67nkxd2nqqqd 1308088852 4202046543 0 11 1 57 1 49 g67nkxd2nqqqd 1308088852 1991738870 0 11 1 39 1 38 g67nkxd2nqqqd 2154937993 1991738870 1 11 27 39 72 38 g67nkxd2nqqqd 2154937993 4202046543 1 11 27 57 72 49 92 rows selected. Elapsed: 00:00:02.53 SQL> 36 © 2009/2010 Pythian
  • 37.
    SQL Plan flippingExample (3) … • In the result … • Load on the system was reduced by 5 times • Takeaways … A. SQL Plans may flip from good plans to … B. SQL Outlines/Profiles may help some times C. AWR provides good input for such analysis • Why SQL Plans may flip? 1. Bind variable peeking / adaptive cursor sharing 2. Statistics change (including difference in partitions stats) 3. Adding/Removing indexes 4. Session/System init.ora parameters (nls_sort/optimizer_mode) 5. Dynamic statistics gathering (sampling) 6. Profiles/Outlines/Baselines evolution 37 © 2009/2010 Pythian
  • 38.
    Conclusions … • AWR = DBA_HIST% views ( snapshots from V$% views ) • Sometimes it is the only source of information • AWR contains much more information that default AWR reports and Grid Control could provide you • Be careful mining data (there are some gotchas) • Don’t be afraid to discover/mine the AWR data I can show you the door … … but it is you who should walk through it 38 © 2009/2010 Pythian
  • 39.
    Pythian Facts • Founded in 1997, over 14 years • 100+ DBAs ! (140 employees) • 5 offices in 5 countries (true follow the sun model) • Employ • 6 Oracle ACEs (Including 1 ACE director) • Several Oracle Masters • Plenty of technical geeks • Platinum level partner in the Oracle Partner Network • Actively supports technical communities via • Blogging • Conferences • SIGs and other events 39 © 2009/2010 Pythian
  • 40.
    Google Yury Oracle[LinkedIn, twitter, blog, email, mobile, …] Additionalget the presentation Email me to Resources • www.oracle.com/scan • www.pythian.com/exadata • www.pythian.com/news/tag/exadata - Exadata Mission Blog Let you remember/consider AWR • www.pythian.com/news_and_events/in_the_news Article: “Making the Most time youExadata” next of Oracle troubleshoot Performance issue! My Oracle Support notes 888828.1 and 757552.1 Thank you! Pythian •Like Pythian on facebook: http://on.fb.me/pythianfacebook •Follow us on LinkedIn: http://linkd.in/pythian •Follow Pythian on Twitter @pythian (www.twitter.com/pythian) 40 © 2009/2010 Pythian

Editor's Notes

  • #8 Trace Analyzer, also known as TRCANLZR or TRCA, is a tool provided by Oracle Server Technologies Center of Expertise - ST CoE. TRCA inputs one or several SQL trace(s) generated by Event 10046 and outputs a diagnostics report in two formats (html and text). These reports are commonly used to diagnose processes performing poorly.
  • #12 REM File: yury1.sqlset echo off feed off lines 110 pages 9990REM Copyright (c) 2011, TPG, all rights reservedclear breakclear colclear computettitle offbtitle offundefine starthh24undefine endhh24undefinedaysagocol beginttim heading &apos;Begin|Interval&apos; format a15 trunccol numtimes format 9,990 head &apos;Num|Times&apos;col sql_id heading &apos;SQL ID&apos; format a13col sumcputimeDelta heading &apos;SUM|CPU|Time|Delta&apos; format 9,999,999,990col buffer_gets_Delta heading &apos;Buff|Gets&apos; format 999,999,990col sumdiskreadsDelta heading &apos;SUM|Disk|Reads&apos; format 9,999,990col sumiowaitDelta heading &apos;SUM|IO|Wait&apos; format 9,999,999,990break on beginttim skip 1ttitle skip 1 center &apos;Top 10 SQL Delta Statistics &amp;&amp;daysago Days ago Hours &amp;&amp;starthh24 to &amp;&amp;endhh24&apos; skip 2spool yury1.lisselect * from( selects.SQL_ID , sum( s.CPU_TIME_DELTA ) sumCPUTIMEDELTA , sum( s.DISK_READS_DELTA ) sumDISKREADSDELTA , sum( s.IOWAIT_DELTA ) sumiowaitDELTA , count( * ) numtimes from DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p, DBA_HIST_SQLTEXT t where ( s.SNAP_ID = p.SNAP_ID ) and ( s.SQL_ID = t.SQL_ID ) and ( EXTRACT( HOUR FROM p.END_INTERVAL_TIME ) between &amp;starthh24 and &amp;endhh24 ) and ( t.COMMAND_TYPE != 47 ) -- Exclude PL/SQL blocks from output and ( p.END_INTERVAL_TIME between ( SYSDATE - &amp;daysago ) and SYSDATE ) group bys.SQL_ID order by sum( s.CPU_TIME_DELTA ) desc )whererownum &lt; 11/PROMPTspool offclear breakclear colclear computettitle offbtitle offset echo on feedback on lines 80 pages 60 term on
  • #24 CREATE OR REPLACE function phy_get_secs (t1 in timestamp, t2 in timestamp)return number is sec_diff number;t_delta interval day to second;begint_delta := t2 - t1;sec_diff := ( extract(hour from t_delta)*60*60 + extract(minute from t_delta)*60 + extract(second from t_delta));return sec_diff;end;/show err
  • #26 set pages 1000prompt SNAP_TIME_P|SNAP_TIME|SEC_D|VALUE_P|VALUE|VALUE_D|VALUE_PER_SECselect /*+ ORDERED FULL(st) */ LAG (st.SNAP_TIME) OVER (ORDER BY st.SNAP_TIME)||&apos;|&apos;||st.SNAP_TIME||&apos;|&apos;||trunc((st.SNAP_TIME - LAG (st.SNAP_TIME) OVER (ORDER BY st.SNAP_TIME))*24*60*60)||&apos;|&apos;|| LAG (ss.value) OVER (ORDER BY st.SNAP_TIME)||&apos;|&apos;||ss.value||&apos;|&apos;|| ( ss.value - LAG (ss.value) OVER (ORDER BY st.SNAP_TIME) )||&apos;|&apos;||trunc( ( ss.value - LAG (ss.value) OVER (ORDER BY st.SNAP_TIME) ) / ((st.SNAP_TIME - LAG (st.SNAP_TIME) OVER (ORDER BY st.SNAP_TIME))*24*60*60) ) fromv$databasevd,stats$snapshotst, V$STATNAME sn, STATS$SYSSTAT ss where 1=1 and ss.snap_id=st.snap_id and ss.instance_number=1 and ss.dbid=vd.dbid and sn.STATISTIC#=ss.STATISTIC# and sn.name=&apos;parse count (hard)&apos; and st.SNAP_TIME between trunc(sysdate)-14 and trunc(sysdate) order by st.SNAP_TIME;select s.BEGIN_INTERVAL_TIME,s.END_INTERVAL_TIME, t.VALUE EVALUE, LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) BVALUE,t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) DVALUE,s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME DTIME, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) H, EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) M, EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) S, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) Secs, (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/(EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)) VAL_SECfrom DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT twhere 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = &apos;parse count (hard)&apos;order bys.BEGIN_INTERVAL_TIME;CREATE OR REPLACE function phy_get_secs (t1 in timestamp, t2 in timestamp)return number is sec_diff number;t_delta interval day to second;begint_delta := t2 - t1;sec_diff := ( extract(hour from t_delta)*60*60 + extract(minute from t_delta)*60 + extract(second from t_delta));return sec_diff;end;/show errcolumn snap_date for a20select cast (s.BEGIN_INTERVAL_TIME as date) snap_date, (t.TIME_WAITED_MICRO-LAG (t.TIME_WAITED_MICRO) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/(t.TOTAL_WAITS-LAG (t.TOTAL_WAITS) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/1000 ms_per_ev, (t.TOTAL_WAITS-LAG (t.TOTAL_WAITS) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/(EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)) ev_SECfrom DBA_HIST_SNAPSHOT s, DBA_HIST_SYSTEM_EVENT twhere 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.EVENT_NAME = &apos;db file scattered read&apos;order bys.BEGIN_INTERVAL_TIME;select sh.BEGIN_INTERVAL_TIME d ,trunc(st.TOTAL_WAITS-(LAG(st.TOTAL_WAITS) OVER (ORDER BY sh.BEGIN_INTERVAL_TIME))) tw_d ,trunc(st.TOTAL_TIMEOUTS-(LAG(st.TOTAL_TIMEOUTS) OVER (ORDER BY sh.BEGIN_INTERVAL_TIME))) tt_d ,trunc((st.TIME_WAITED_MICRO-(LAG(st.TIME_WAITED_MICRO) OVER (ORDER BY sh.BEGIN_INTERVAL_TIME)))/1000000) w_d_secsfrom DBA_HIST_SYSTEM_EVENT st, DBA_HIST_SNAPSHOT shwhere 1=1and st.SNAP_ID=sh.SNAP_IDand st.EVENT_NAME = &apos;enq: TX - row lock contention&apos;order by sh.BEGIN_INTERVAL_TIME;
  • #28 select s.BEGIN_INTERVAL_TIME,s.END_INTERVAL_TIME, t.VALUE EVALUE, LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) BVALUE,t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME) DVALUE,s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME DTIME, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) H, EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) M, EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) S, EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) Secs, (t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/(EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)) VAL_SECfrom DBA_HIST_SNAPSHOT s, DBA_HIST_SYSSTAT twhere 1=1 and s.SNAP_ID = t.SNAP_ID and s.DBID = t.DBID and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE) and s.DBID = (select DBID from V$DATABASE) and t.STAT_NAME = &apos;parse count (hard)&apos;order bys.BEGIN_INTERVAL_TIME;CREATE OR REPLACE function phy_get_secs(t_delta interval day to second)return number is sec_diff number;begin-- t_delta:= t2 - t1;sec_diff := ( extract(hour from t_delta)*60*60 + extract(minute from t_delta)*60 + extract(second from t_delta));return sec_diff;end;/show err
  • #41 DBMS_XPLAN.DISPLAY_AWR