This document discusses maximizing database performance through performance tuning with DB Time. It introduces DB Time as the fundamental measure of Oracle performance, including CPU time and wait time spent in database calls. The document outlines how tools like the Automatic Database Diagnostic Monitor (ADDM) and Enterprise Manager use DB Time-based methods to identify performance issues and recommend tuning solutions by analyzing where the most DB Time is being spent. It presents DB Time as the common performance metric used across Oracle tools and recommends following the DB Time by investigating the top wait events and SQL statements consuming the most time.

1. 1

2. <Insert Picture Here>
Maximizing Database Performance: Performance Tuning with DB
Time
Kurt Engeleiter, John Beresniewicz, Cecilia Gervasio
Oracle America

3. 3
The following is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions.
The development, release, and timing of any
features or functionality described for Oracle‟s
products remains at the sole discretion of Oracle.

4. 4
Agenda
• NEWS: DB Time is not news
• DB Time Fundamentals
• DB Time Method for Performance Tuning
• DB Time Based Tools
– ADDM
– Enterprise Manager

5. 5
<Insert Picture Here>
NEWS: DB Time is Not News

6. 6
DB Time is Pervasive, Familiar, Evolutionary
• DB Time is the fundamental measure of Oracle
performance
– Arrived at progressively but designed-in from 10g
• You are likely using DB Time methods already
– AWR and ASH reports
– Enterprise Manager
• DB Time is performance analysis science matured
– Wait event analysis, YAPP and tracing are all its ancestors

7. 7
DB Time on the Database Home Page
• Host CPU
• Is host CPU-bound? Is this DB responsible?
• Active Sessions
• DB Time accumulation by CPU, User I/O, Wait

8. 8
DB Time on the Performance Page
• Area under curve is equal to DB time
• Wait class breakdown indicates solution space

9. 9
DB Time on the Top Activity Page
DB Time
• Plots DB Time from ASH data
• Shows top DB Time consumers by SQL and Session

10. 10
DB Time in AWR Report
• Time Model measures and
accumulates DB Time through
session instrumentation
• DB Time and DB CPU most
important
• Also shows DB Time expended
in various sub-operations

11. 11
DB Time in AWR Report
• If you tune by Wait
Events you are tuning by
DB Time
• Top 5 Timed Events in
the AWR report shows
CPU time and non-idle
waits for foregrounds

12. 12
DB Time in the Grid Control DB Loadmap
• Maps DB Time across the enterprise
• Cell size proportional to DB Time

13. 13
<Insert Picture Here>
DB Time Fundamentals

14. 14
Why Do We Care About Time?
• Human time is critical to the enterprise
• Systems performance affects business goals
– Human time + technology resource time
• “Time is money”
• Performance improvement means doing things faster
Performance is always and only about time

15. 15
DB Time Defined
• Total time in database calls by foreground sessions
• Includes CPU time, IO time and non-idle wait time
• DB Time is not called response time on purpose
• Common currency for Oracle performance analysis
Database time is total time spent by user
processes either actively working or actively
waiting in a database call.

16. 16
Active Session =
Session currently spending time in a database call
Database Time (DB Time) =
Total time session spends in all database calls
Average Activity of the Session (% Activity)
= Ratio of time active to total wall clock time
= Sum(Active Time) / Elapsed Time
Browse
books
Read reviews
for a book Add to cart Checkout
TIME
= time spent in database
Session-level Concepts

17. 17
= time spent in database
Multiple Sessions
TIME
User 1
User 2
User 3
User n
DB Time = Sum of DB Time over all sessions
Avg. Active Sessions = Sum of Avg. Activity over all sessions
= Sum(DB Time) / Elapsed Time

18. 18
TIME
User 3
User 2
User 1
User n
t0 t1
1
2
3
4
Active Sessions over time
Visualizing DB Time

19. 19
Enterprise Manager Performance Page
• Active Sessions by wait class over time
• Colored area is the amount of DB Time
DB Time on User I/O is always blue
DB Time on CPU is always green

20. 20
<Insert Picture Here>
DB Time and
System Performance

21. 21
System Load and DB Time
• More users
More database calls
DB Time increases
• Larger transactions
More SQLs per call
DB Time increases
DB time increases as system load increases.

22. 22
System Performance and DB Time
• I/O performance degrades
I/O wait time increases
DB Time increases
• Application performance degrades
Longer elapsed time per SQL
DB Time increases
DB time increases as performance degrades.

23. 23
Host Performance and DB Time
• Host is CPU bound
Foregrounds accumulate active run-queue time
Wait event times are artificially inflated
DB Time increases
Tune for CPU before waits when CPU constrained

24. 24
CPU Run-Queue and DB Time
Db file sequential read
Run-queue
On CPU Db file sequential read
Run-queue
On CPU
DB time is inflated when host is CPU-bound
User 1
Actual wait time Actual wait time
Recorded wait time Recorded wait time

25. 25
<Insert Picture Here>
The DB Time Method for
Performance Tuning

26. 26
The DB Time Method: Short Course
Determine where database
time is spent, and reduce it!
or
just ask ADDM

27. 27
The DB Time Method: Process
1. Identify performance issue
2. Scope the issue
3. Set goals
4. Data capture
5. Investigate DB time distribution
– Identify largest potential improvement
6. Modify system to tune for largest gain
7. Evaluate against goals
Performance tuning by removing excess DB time
AWR handles this

28. 28
Identify and Scope the Problem
• Big spike = big problem!
• Container (host) or database issue?

29. 29
Investigate DB Time Distribution
• Identify excess DB time
– Look for large skews in time that may represent inefficiencies
• System scope:
– Resource limits – is problem outside the DB?
– Resource contention (latches)
• Application scope:
– Service, module, action
– SQLID, rowsource
– Resource contention (enqueues)
• Session scope:
– Long-running SQL
– Resource contention (holding enqueues in uncommitted txns)

30. 30
Investigate DB Time distribution
• Which wait classes are dominant?
• “Top” lists show skew in key dimensions (SQL, session, file, etc.)

31. 31
Identify Time-saving Modifications
• System scope:
– Initialization parameter changes
– Add resources or capacity
• Application scope:
– SQL Tuning Advisor: SQL profiles
– Real-time SQL Monitoring
– Access Advisor: Physical schema layout
– Re-write SQL
• Session scope:
– Kill session for immediate contention relief
– Application design changes often long-term solution

32. 32
Critical Success Factors
• Match solution scope to problem scope
– Don‟t tweak optimizer parameters before tuning SQL
• Address root causes, not symptoms
• Proceed iteratively one fix at a time
– Concurrent fixes should be orthogonal
• Measure and validate results at each successive step
• Stop when goals are met

33. 33
Advantages of the DB Time Method
• Tunes exactly what affects users:
– Time wasted in database
• Data capture not necessary:
– „Always on‟ instrumentation framework (AWR, ASH)
– No requirement to reproduce problem (unlike tracing)
• Superior coverage of all scope levels
– Works for concurrency problems such as locking
– Tight integration with kernel-engineered tools
• Maturation and extension of well-proven methodologies

34. 34
<Insert Picture Here>
DB Time Based Tools:
ADDM
Enterprise Manager

35. 35
Tools for Applying DB Time Method
Two use-cases, one method:
1. Tuning steady-state performance
– Improve overall workload throughput or response time
– Best practice: use ADDM
2. Diagnosing transient performance problems
– Confirm and investigate reported performance issues
– Best practice: use EM real-time screens

36. 36
Example: Using ADDM
• Users complain about bad performance
• AWR report indicates some unusual issues on system
• What to do?

37. 37
What does the Internet advise?
• Let‟s search on „enq: HW contention‟
• 9,000 hits and LOTS of discussion

38. 38
Example: Following the links
• Many hits are other people asking the same question – what do I
do about this wait event?

39. 39
Solution: ADDM in Enterprise Manager
• Performance Page shows a spike of unusual activity
• Click on the ADDM analysis icon

40. 40
Solution: Using ADDM
• ADDM performance findings are DB Time based
• Ranked by impact = percent of DB Time affected
• Click on the issue with the highest impact

41. 41
Solution: Using ADDM
• ADDM recommendations describe solutions for findings
• EM presents „implement‟ actions where possible

42. 42
Best Practice: Enterprise Manager Real-time Interface
• Transient (sub-hour) or immediate time scope
– Requires interactivity of UI
• „Click on the big stuff‟
– Data visualizations display skew directly
• Takes some expertise to separate symptoms from
root causes

43. 43
Click on the Big Stuff – Follow the DB Time
• Spike in DB Time signals possible performance issue
• Click through to Top Activity for more details

44. 44
Follow the DB Time – Top Activity Drilldown
• Top Activity Page confirms problem and shows highly skewed DB
Time by SQL ID. Click on top SQL ID for further details.

45. 45
Click on the Big Stuff – Follow the DB Time
• SQL Details confirms this SQL produced the spike in DB Time and
shows parallel execution. Click on „SQL Monitoring‟ to see DB
Time by execution plan steps.

46. 46
Follow the DB Time – SQL Execution Details

47. 47
DB Time Method Summary
• DB time is the fundamental performance metric
• The method allows DB time analysis at many scopes
– Proper scoping of problems and solutions is critical to success
• DB time based diagnosis removes value judgments
– Scientific method, not sorcerer‟s magic
Performance improvement means doing the same
work in less DB Time

49. 49
Oracle Enterprise Manager 11g
Resource Center
Access Videos, Webcasts, White Papers, and More
Oracle.com/enterprisemanager11g

50. 50
Additional Oracle Enterprise Manager Sessions
Wednesday, Sept. 22 Location
• 10:00 a.m. - Manage the Manager: Diagnosing and Tuning
Oracle Enterprise Manager
• Moscone S Rm 102
• 11:30 a.m. - Maximizing Database Performance: Performance
Tuning with DB Time
• Moscone S Rm 104
• 11:30 a.m. - Make Upgrades Uneventful Using Oracle
Enterprise Manager and My Oracle Support
• Moscone S Rm 310
• 12:30pm – Extracting Real Value from Your Data with Apache
Hadoop
• Hilton Hotel, Plaza B
• 1:00 p.m.- Reducing the Risk of SOA Transactions
• Marriott Marquis, Salon
6
• 1:00 p.m. - SQL Tuning Roundtable with Oracle Gurus • Moscone S Rm102
• 4:45 p.m.- Strategies for Monitoring Large Datacenters with
Oracle Enterprise Manager
• Moscone S Rm102
• 4:45 p.m.- Oracle SOA Management Best Practices, Tips, and
Techniques
• Moscone W L3, Rm
3018
• 4:45 p.m.- Oracle E-Business Suite Technology: Vision,
Release Overview, Product Roadmap
• Moscone W L3, Rm
3002 / 3004

51. 51
Additional Oracle Enterprise Manager Sessions
Thursday, Sept. 23 Location
• 9:00 a.m. - Oracle WebLogic Server Management for Oracle
DBAs
• Marriott Marquis,
Salon 9
• 9:00 a.m. - Enabling Database as a Service Through Agile Self-
Service Provisioning
• Moscone S. Room
102
• 9:00 a.m. - Reduce TCO with Oracle Application Management
Pack for Oracle E-Business Suite
• Moscone W L2, Rm
2024
• 10:30 a.m. - Best Practices for Managing Your PeopleSoft
Applications
• Marriott Hotel, Golden
Gate A
• 10:30 a.m. - Oracle Enterprise Manager Grid Control
Deployment Best Practices
• Moscone S. Room
102
• 10:30 a.m. - Managing Sun SPARC Servers with Oracle
Enterprise Manager Ops Center
• Moscone S. Room
252
• 10:30 a.m. - Heterogeneous Data Masking: Oracle, SQL Server,
and DB2 Database Best Practices
• Moscone S. Room
306
• 12:00 p.m. - Scalable Enterprise Data Processing for the Cloud
with Oracle Grid Engine
• Moscone S. Room
310
• 12:00 p.m. - Spot Problems Before Your Users Call: User
Experience Monitoring for Oracle Apps
• Marriott Hotel, Golden
Gate A
• 12:00 p.m. - Reduce Problem Resolution Time with Oracle
Database 11g Diagnostic Framework
• Moscone S. Room
102

52. 52
Additional Oracle Enterprise Manager Sessions
Thursday, Sept. 23 Location
• 1:30 p.m. - Patching Enterprisewide Databases: Automation
Techniques and Real-World Insights
• Moscone S. Room
310
• 1:30 p.m. - Managing User Experience: Lessons from eBay
• Marriott Hotel, Golden
Gate A
• 1:30 p.m. - Deep Java Diagnostics and Performance Tuning:
Expert Tips and Techniques
• Marriott Marquis,
Salon 9
• 1:30 p.m. - Oracle Enterprise Manager Configuration
Management Unleashed: Top 10 Expert Tips
• Marriott Marquis,
Salon 6
• 1:30 p.m. - Oracle Enterprise Manager Security Best Practices
• Moscone S. Room
102
• 3:00 p.m - The X-Files: Managing the Oracle Exadata and
Highly Available Oracle Databases
• Moscone S. Room
102
• 3:00 p.m. - Monitoring and Diagnosing Oracle RAC
Performance with Oracle Enterprise Manager
• Moscone S. Room
310

53. 53
Oracle Enterprise Manager Hands On Labs
Monday September 20, 2010
3:30 p.m. - 4:30 p.m. Database Performance Diagnostics and Tuning Marriott Hotel,
Salon 12/13, YB Level
5:00 p.m. - 6:00 p.m. Provisioning,Patch Automation, and Configuration Management Pack Marriott Hotel,
Salon 12/13, YB Level
5:00 p.m. - 6:00 p.m. Oracle Application Mgmt. Pack for Oracle E-Business Suite: Monitor/Clone Marriott Marquis, Nob Hill
Tuesday September 21, 2010
11:00 a.m.-12:00 p.m. Using Oracle Application Change Management Pack for Oracle E-Business Suite Marriott Marquis, Nob Hill
12:30 p.m.-1:30 p.m. Database and Application Testing Marriott Hotel,
Salon 12/13, YB Level
2:00 p.m. - 3:00 p.m. Oracle Fusion Middleware Management Marriott Hotel,
Salon 12/13, YB Level
3:30 p.m. - -4:30 p.m. Provisioning,Patch Automation, and Configuration Management Pack Marriott Hotel,
Salon 12/13, YB Level
Wednesday September 22, 2010
4:45 p.m. - 5:45 p.m. Database and Application Testing Marriott Hotel,
Salon 12/13, YB Level
4:45 p.m. - 5:45 p.m. Oracle Application Mgmt. Pack for Oracle E-Business Suite: Monitor/Clone Marriott Marquis, Nob Hill
Thursday September 23, 2010
9:00 a.m. - 10:00 a.m. Database Performance Diagnostics and Tuning Marriott Hotel,
Salon 12/13, YB Level
10:30 a.m. - 11:30 a.m. Oracle Fusion Middleware Management Marriott Hotel,
Salon 12/13, YB Level

54. 54
Oracle Enterprise Manager Demogrounds
D E M O T I T L E L O C A T I O N
Oracle Real Application Testing: Database Replay Moscone West
Oracle Real Application Testing: SQL Performance Analyzer Moscone West
Self-Managing Database: Automatic Performance Diagnostics Moscone West
Self-Managing Database: Automatic Fault Diagnostics Moscone West
Self-Managing Database: Automatic Application and SQL Tuning Moscone West
Application Quality Management: Application Testing Suite Moscone South - S022
Real User Monitoring with Oracle Enterprise Manager Moscone South - S021
Siebel CRM Application Management Moscone South - S024
Real User Monitoring with Oracle Enterprise Manager Moscone West
Oracle WebLogic Server Management and Java Diagnostics Moscone West
SOA Management with Oracle Enterprise Manager Moscone West
Oracle Business Transaction Management Moscone West
Push Button Provisioning and Patch Automation Moscone West
Smart Configuration Management Moscone West
Oracle Enterprise Manager Ops Center Moscone West
Managing the Enterprise Private Cloud Moscone West
System Management, My Oracle Support, and Oracle Enterprise Manager Moscone West
Self Managing Database: Change Management for DBAs Moscone West
Oracle Enterprise Manager: Complete Datacenter Management Moscone West
Self-Managing Database: Data Masking for DBAs Moscone West

55. 55