Oracle's Real Application Testing solution provides a cost-effective and easy-to-use solution for assessing the impact of change on production Oracle databases by capturing and replaying actual database workload and workflows. The solution provides greater business agility and reduces the risk and cost associated with change.
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
Real Application Testing
1. Oracle Database 11g
Real Application Testing
and Enterprise Manager Virtual Classroom
Lynn Ferrante and Michael Whelan
2. Oracle Database 11g
Real Application Testing
and Enterprise Manager Virtual Classroom
Lynn Ferrante and Michael Whelan
3. Database Testing Agenda
• Overview
• Database testing
• Eliminate the risk of change by assessing the impact of
change on production databases
• Database masking
• Eliminate risk of moving data to non-production environments
3
4. Database Testing
• Despite expensive testing success rate low
• Many issues go undetected
• System availability and performance negatively impacted
• Cause of low success rate
• Existing tools provide inadequate testing
• Simulate synthetic workload instead of replaying actual
production workload
• Provide partial workflow coverage
Database Replay makes real-world testing possible
4
6. 1: Install & Setup Test System
Duration (Days)
24 Days
Traditional Approach
Real Application 4 Days
Testing
0
0
0
0
0
0
10
20
30
40
50
60
70
80
90
0
10
11
12
13
14
15
6
7. 2: Understand Application Usage
Duration (Days)
+20 Days
Traditional Approach
Real Application +0 Days
Testing
0
0
0
0
0
0
10
20
30
40
50
60
70
80
90
0
10
11
12
13
14
15
7
8. 3: Identify Key Transactions
Duration (Days)
+20 Days
Traditional Approach
Real Application +0 Days
Testing
0
0
0
0
0
0
10
20
30
40
50
60
70
80
90
0
10
11
12
13
14
15
8
9. 4: Develop Test Scripts
Duration (Days)
+80 Days
Traditional Approach
Real Application +2 Days
Testing
0
0
0
0
0
0
10
20
30
40
50
60
70
80
90
0
10
11
12
13
14
15
9
10. 5: Run Test Workload
Duration (Days)
+5 Days
Traditional Approach
Real Application +5 Days
Testing
0
0
0
0
0
0
10
20
30
40
50
60
70
80
90
0
10
11
12
13
14
15
10
11. Innovate Faster
Duration (Days)
149 Days
Traditional Approach
Real Application 11 Days
Testing
0
0
0
0
0
0
10
20
30
40
50
60
70
80
90
0
10
11
12
13
14
15
11
12. Comparison of LoadRunner & Oracle Real
Application with e-Business Suite
80
80
Time Taken (Days)
60
40
24
20 20
20 5 LoadRunner
4 0 0 2 5 DB Replay
0
Install & Setup Understand Identify Key Generate Run Test
Application Transactions Workload
Usage
Total Testing Time
DB Replay: 2 weeks
“Traditional” Testing: 30 weeks
12
13. Oracle Enterprise Manager
Top-Down Application Management
Application Application
Operations Quality
Management Management
Real User
Experience Monitoring Load Testing and
Accelerators for
Service Level Packaged Apps &
Management Web Services
Application
Packaged Application Testing Suite Functional Testing and
Management Accelerators for
Packaged Apps &
SOA Application Management Web Services
Diagnostic and Tuning Test Management
Provisioning, Configuration
Management Real Application Testing
Change Tracking Data Masking
and Synchronization
13
15. Real Application Testing
• Value
• Rapid technology adoption Deploy
• Higher testing quality
• Business Benefit Test
Change
• Lower cost
• Lower risk
Remediate
15
16. ROI
Ovum Summit
Oracle Real Application Testing – business agility
through superior testing
http://www.oracle.com/corporate/analyst/reports/infrastructure/dbms/ovum
-real-application-testing.pdf
16
17. Real Application Testing
Identify SQL statement with Captures a production
performance regressions workload and allows replay of
caused by system changes the workload on a test system
Execute SQL Execute SQL
serially SQL concurrently
DATABASE
PERFORMANCE
REPLAY
ANALYZER
17
19. Database Replay
• Replay actual production database workload in test
environment
• Identify, analyze and fix potential instabilities before making
changes to production
• Capture Workload in Production
• Capture full production workload with real load, timing &
concurrency characteristics
• Move the captured workload to test system
• Replay Workload in Test
• Make the desired changes in test system
• Replay workload with full production characteristics
• Honor commit ordering
• Analyze & Report
• Errors
• Data divergence Analysis & Reporting
• Performance divergence
19
21. Common Usage Scenarios
• Database initialization/config parameter changes
• Schema changes
• Changes to operating systems and hardware
• Gathering statistics
• Database upgrades including patch deployments
• Implementation of tuning recommendations
• Partitioning
• Compression
• etc
21
22. Database Replay Workflow
Production (10.2.0.4) Test (11.1)
Clients
… Replay Driver
Mid-Tier
…
Storage
Storage
Analysis &
Capture Process Replay
Reporting
22
23. Step 1: Workload Capture
Production System
• All external client requests
captured in binary files
Client Client Client
• System background, internal …
activity excluded
File System
• Minimal performance
overhead for capture Middle Tier
• For RAC, shared and local File 1
file system supported
File 2
• Specify interesting time
period for capture, e.g., peak …
workload, month-end
processing, etc. File n
Storage
• Can capture on 10.2.0.4 and
replay on 11g
23
24. Step 2: Process Workload Files
• Setup test system
Test System
• Test DB is at same point in time
as before production capture
• Use RMAN to physically restore
production db from backup File 1
File 1
• Use Snapshot standby
File 2
• Use imp/exp, Data Pump, etc.
File 2
…
• Processing transforms captured
data into replayable format … File n
• Once processed, workload can be File n
replayed many times
• For RAC copy all capture files to Metadata
single location for processing Replay Files
Capture Files
25
25. Step 3: Replay Workload
Test System
• Replays workload preserving
timing, concurrency and
dependencies of the capture
system Replay Driver
• Replay Driver is a special
client program that
consumes processed File 1
workload and sends requests
to the replay system File 2
…
• Replay Driver consists of one
or more clients. For File n
workloads with high
concurrency, it may be Metadata
necessary to start multiple Replay Files
clients to drive workload
26
26. Analysis & Reporting
• Comprehensive reports are provided for analysis purposes
• There (3) types of divergences are reported
• Data Divergence: Number of rows returned by each call are
compared and divergences reported
• Error Divergence: For each call error divergence is reported
• New: Error encountered during replay not seen during capture
• Not Found: Error encountered during capture not seen during
replay
• Mutated: Different error produced in replay than during capture
• Performance Divergence
• Capture and Replay Report: Provides high-level performance
information
• ADDM Report: Provides in-depth performance analysis
• AWR, ASH Report: Facilitates comparative or skew analysis
27
29. SPA Benefits
• Enables identification of SQL performance regressions before
end-users can be impacted
• SPA can help with any change that impacts SQL execution plan
• DB upgrades
• Optimizer statistics refresh
• New indexes, Materialized Views, Partitions, etc.
• Automates SQL performance tracking of hundreds of thousands of
SQL statements – impossible to do manually
• Captures SQL workload with low overhead
• Integrated with SQL Tuning Advisor and SQL Plan Baselines for
regression remediation
30
30. Production Test
Client Client Client
…
Middle Tier Re-execute SQL Queries
Capture SQL
… … Use SQL Tuning
Oracle DB Advisor to tune
regression
Storage
31
31. SQL Performance Analyzer: Workflow
Production Test
Make Change
Steps (1) (2) (3) (4) (5)
Capture Transport Execute SQL Execute SQL Compare
SQL (STS) STS Pre-change Post-change Perf.
(6) Reiterate
No
Done?
(7)
Yes
Tuned System
Production Change /
(7)
Tuning Deployment
32
32. Step 1: Capture SQL Workload
• SQL Tuning Set (STS) used to store SQL
workload
• STS includes:
• SQL Text
• Bind variables
• Execution plans
Cursor Cache
• Execution statistics
Incremental Capture • Incremental capture used to populate
STS from cursor cache over a time
period
• SQL tuning set’s filtering and ranking
SQL Tuning Set capabilities filters out undesirable SQL
• SQL workload captured in 10.2.0.1 and
Production Database higher can be used for SPA tasks in 11g
33
33. Step 2: Move SQL Workload to Test System
Cursor Cache
SQL Tuning Set
Export/Import SQL Tuning Set
Production Database Test Database
• Use Enterprise Manager
34
34. Step 3: Execute SQL Before Making Change
SQL Tuning Set • Establishes SQL workload performance
baseline
Fetch Next SQL
• SQL execution plan and statistics
captured
Test Execute • SQL executed serially (no concurrency)
Execution Plan & • Each SQL executed only once
Statistics
Save Result • DDL/DML skipped
• Option to do Explain Plan only analysis
SQL Performance Analyzer
35
35. Step 4: Execute SQL After Making Change
SQL Tuning Set
Fetch Next SQL • Manually implement the planned change
• Database upgrade, patches
• Optimizer statistics refresh
• Schema changes
Test Execute • Database parameter changes
• Tuning actions, e.g., SQL Profile creation
Execution Plan &
Statistics
Save Result
• Re-execute SQL after change
Completed • Gathers new SQL execution plans
and statistics
SQL Performance Analyzer
36
36. Step 5: Compare & Analyze Performance
Completed Completed
• Compare performance using
different metrics, e.g.,
• Elapsed Time
• CPU Time
• Optimizer Cost
• Buffer Gets
Compare
SQL Performance
• SPA Report shows impact of
change for each SQL
• Improved SQL
Analysis Report • Regressed SQL
• Unchanged SQL
• Fix regressed SQL using SQL
Tuning Advisor or SQL Plan
Baselines
SQL Performance Analyzer
37
40. Real Application Testing Summary
• Provides cost-effective and easy-to-use solution of assessing
impact of change on production systems
• Holistic and actual workload testing results in lower risk
• Brings down testing cycle from months down to days
• Lowers hardware costs by eliminating need for mid-tier and
application setup on test system
• Maximizes ROI by leveraging Diagnostics and Tuning Pack to
remediate regressions
• With Real Application Testing businesses can
• Stay competitive
• Improve profitability
• Be compliant
41
41. Real Application Testing: Tools of the Trade
SQL Performance Analyzer Database Replay
What is it? • Predicts SQL performance deviations • Replays real database workload
before end-users can be impacted, on test system, helps assess
helps assess impact of change on impact of change on workload
SQL response time throughput
How it works? • Executes each SQL, stored in SQL • Captures workloads and replays it
Tuning Set, in isolation using with production characteristics
production context and then including concurrency,
compares before and after execution synchronization &
plans and run-time statistics dependencies
When to use? • Unit testing of SQL with the goal to • Comprehensive testing of all
identify the set of SQL statements sub-systems of the database
with improved/regressed performance server using real production
workload
SQL SQL Concurrency
Dependency Speed up/down
42
43. Database Diagnostic Pack
Introduction
• Part of the database management pack family
• Provides unique functionalities
• Industry’s first performance self-diagnosing engine
• Lightweight, Automatic, Out-of-box performance data
warehouse
• Guided problem resolution
• Introduces a new, easy to use performance tuning methodology
• Not just another tool
• Core functionality built right into the database server
• Makes it most efficient, integrated and accurate
44
44. Database Diagnostic Pack
Key Features
• Automatic Workload Repository (AWR)
• Active Session History (ASH)
• Automatic Database Diagnostic Monitor (ADDM)
45
45. How Does ADDM Work?
• Top Down Analysis Using
Snapshots in AWR Snapshots
Automatic Workload
Repository • Throughput centric - Focus on
reducing time ‘DB time’
Automatic Diagnostic Engine
Self-Diagnostic Engine • Classification Tree - based on
decades of Oracle
performance tuning expertise
• Real-time results
• Don’t need to wait hours to
see the results
• Pinpoints root cause
High-load IO / CPU
RAC issues • Distinguishes symptoms
SQL issues
from the root cause
• Reports non-problem areas
System Network +
SQL
Advisor
Resource DB config • E.g. I/O is not a problem
Advice Advice
46
46. Database Tuning Pack
Introduction
• Part of the database management pack family
• Provides unique, automatic and deterministic SQL
tuning functionality
• Core functionality built in the Oracle Database 10g
kernel
• SQL Tuning Advisor
• SQL Access Advisor
• Requires diagnostic pack
47
47. SQL Profiles
Contains auxiliary information specific to a given SQL
statement
– Customized optimizer settings
Based on past execution history
– Compensation for missing or stale statistics
– Compensation for errors in optimizer estimates
Estimation errors occur due to data skews and correlations,
complex filters and joins
Doesn’t freeze the execution plan
Doesn’t require any change to the SQL text
– Ideal for Packaged Apps
Persistence: Works across shutdowns & upgrades
Transportable across databases (10.2)
48
49. Oracle Data Masking
Data Masking for Non-Production Environments
• Eliminate risk of moving data to
non-production environments LAST_NAME SSN SALARY
AGUILAR 203-33-3234 40,000
• Share data with partners BENSON 323-22-2943 60,000
without exposing sensitive data
Production
• No impact on production
environments
• Sensitive data never leaves the
database (masked in place) Non-Production
• Automated masking process LAST_NAME SSN SALARY
ANSKEKSL 111—23-1111 40,000
preserves application integrity BKJHHEIEDK 111-34-1345 60,000
50
50. Summary
• Real Application Testing for managing risk
• Provides cost-effective and easy-to-use solution of assessing
impact of change on production systems
• Lowers hardware costs by eliminating need for mid-tier and
application setup on test system
• Maximizes ROI by leveraging Diagnostics and Tuning Pack to
remediate regressions
• Diagnostic and Tuning Pack for proactive
management
• Industry’s first performance self-diagnosing engine
• Data Masking for security of data
51
51. Comparison of Traditional Approach &
Oracle Real Application with e-Business
Suite
80
80
Time Taken (Days)
60
40
24
20 20
20 5 LoadRunner
4 0 0 2 5 DB Replay
0
Install & Setup Understand Identify Key Generate Run Test
Application Transactions Workload
Usage
Total Testing Time
DB Replay: 2 weeks
“Traditional” Testing: 30 weeks
52
