Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Proof of Concept with Real Application Testing 12c

2,559 views

Published on

Evaluate how certain real world database workload behaves on different I/O subsystem, processors and
architecture or the coexistence with other databases is the goal of a Proof of Concept. The need of testing
real production workloads to eliminate uncertainty with help of techniques like Workload Folding, Time
Shifting and Schema Remapping, this talk will produce evidence that exploring Real Application Testing
features in 12c leverage what can be accomplished by a Proof of Concept.

Published in: Technology
  • Be the first to comment

Proof of Concept with Real Application Testing 12c

  1. 1. PROOF OF CONCEPT with Real Application Testing 12c Luís Marques - @drune - http://lcmarques.com
  2. 2. Me Name: Luís Marques http://lcmarques.com / @drune / lcarapinha@gmail.com Luís Marques - @drune - http://lcmarques.com
  3. 3. We will discuss Database Replay features on 12c version and how we can apply stress test and workload scale-up techniques to ensure a real world proof of concept SQL Perfomance Analyzer Actual PoC results We will not to discuss Luís Marques - @drune - http://lcmarques.com
  4. 4. PoC – What really is? An huge amount of work to plan and execute (and re- execute until the end of available time period) Most work is for DBA if Real Application Testing is used. Luís Marques - @drune - http://lcmarques.com
  5. 5. The (real) way it was… Hardware arrives Setup required Recent Backup required Restore your DB into new hardware Flashback Guaranteed Restore PointSetup Application: - New hardware - Install and Configure: - AS, Clients, Network, etc We are ready! 2 weeks of infinite fun and infinite problems Luís Marques - @drune - http://lcmarques.com
  6. 6. The (real) way it was… Luís Marques - @drune - http://lcmarques.com Database scheduler Jobs created on-the-fly Small reference in docs (not even in restrictions area)
  7. 7. The way it should be Working with the vendor to provide support for Real Application Testing Client requested a simple and effective way to test new platform and software releases Luís Marques - @drune - http://lcmarques.com
  8. 8. The way it should be Hardware arrives Setup required Recent Backup required Restore your DB into new hardware using START_SCN from Capture Flashback Guaranteed Restore Point DB Replay DB Capture Luís Marques - @drune - http://lcmarques.com
  9. 9. Database Replay - Capture Luís Marques - @drune - http://lcmarques.com
  10. 10. Before start any capture… RAT Patches on Production and PoC database (Doc ID 560977.1) Capture Restrictions: direct path load external files with SQL*Loader, scheduler jobs, flashback querys, distributed trx, etc.. Workload is important: sysdate use,avoid maintenance windows or EM activity No ongoing transactions during capture - Plan you database shutdown and start capture immediately after startup Luís Marques - @drune - http://lcmarques.com
  11. 11. Workload Capture Payment processing starts at 19:00: 18:35 – SQL> shutdown immediate 18:45 – SQL> startup restrict 18:50 – $ sqlplus / as sysdba 18:50 – SQL> @capture_payment_day.sql DBMS_WORKLOAD_CAPTURE.ADD_FILTER ('PaymentUser', 'USER’, 'PAY_A'); DBMS_WORKLOAD_CAPTURE.START_CAPTURE (name =>'capture_payment_day', dir => 'DB_REPLAY_DIR', auto_unrestrict => TRUE, default_action => 'EXCLUDE', sts_cap_interval => 300); Luís Marques - @drune - http://lcmarques.com
  12. 12. Workload Capture Populate DBA_WORKLOAD_CAPTURES / FILTERS SQL> exec dbms_workload_capture.get_capture_info('DB_REPLAY_DIR'); Save the START_SCN to use with RMAN UNTIL SCN to restore your database on PoC database. Luís Marques - @drune - http://lcmarques.com
  13. 13. Workload Capture Export AWR data to compare performance between Production and PoC database: SQL> exec dbms_workload_capture.export_awr(2); Dump of AWR will be used after replay to compare AWR on PoC database AWR data Luís Marques - @drune - http://lcmarques.com
  14. 14. Workload Capture – The intelligent stuff Workload Intelligence – Java programs to analyze Captured data Don’t run it on production system (Heavy memory consumption) Allow you to know more about the capture by patterns Create user wi Create Workload Job Create Workload Model Identify Patterns Generate Report Java –classpath … oracle.dbreplay.workload.intelligence.LoadInfo -job wi_job1 -cdir /home/oracle/scripts/rat/db_replay_dir -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi Java –classpath … oracle.dbreplay.workload.intelligence.BuildModel -job wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi Java –classpath … oracle.dbreplay.workload.intelligence.FindPatterns -job wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi Java –classpath … oracle.dbreplay.workload.intelligence.GenerateReport -job wi_job1 -cstr jdbc:oracle:thin:@baco:1521:bacodb2 -user wi –top 5 –out intel_report.html Luís Marques - @drune - http://lcmarques.com
  15. 15. How an intelligence report looks… Pattern is a sequence of transactions that run under “certain order”. Allow you to identify mostly used “code paths” – Example: create a customer or process a visa card “code paths” Patterns are ordered by different columns (DB Time, Nr. Executions and Length) Pattern id = 2  Create a customer run 2333 times during the capture with a DB Time 9 seconds (0,78% of total DB Time) Luís Marques - @drune - http://lcmarques.com Pattern id = 2 Pattern id =1
  16. 16. Database Replay – Replay Luís Marques - @drune - http://lcmarques.com
  17. 17. Before start any replay… Setup guarantee restore points before any replay Run java utility “Workload Analyzer” – Check the quality of capture (insufficient data, errors, etc) Run a small test replay (~ 20 min) before full replay - Avoid potential issues with full replay Set up your system date to the same time as time of capture start if application relies on sysdate Before performance analysis perform a replay analysis and review divergences Luís Marques - @drune - http://lcmarques.com
  18. 18. Before start replay - sysdate Set system clock on PoC system to the start of time of capture Make sure that NTP daemon is disabled Make sure that AWR snapshots are still created automatically after system clock changed Restart PoC database after change in the clock. Luís Marques - @drune - http://lcmarques.com
  19. 19. Before start replay – The Workload Analyzer Java utility provided by Oracle to analyze the quality of your capture. $ java -classpath $ORACLE_HOME/jdbc/lib/ojdbc6.jar:$ORACLE_HOME/rdbms/jlib/dbrparser.jar:$ORACLE_H OME/rdbms/jlib/dbranalyzer.jar: oracle.dbreplay.workload.checker.CaptureChecker /home/oracle/scripts/rat/db_replay_dir jdbc:oracle:thin:@baco.localdomain:1521:prodpay2 Almost all sessions have been captured in-flight. PL/SQL blocks or functions have 'complicated' logic or multiple commits in them, they are hard to synchronize and they behavior might change during replay. wcr_cap_analysis.html Luís Marques - @drune - http://lcmarques.com
  20. 20. Captured workload: Sequential payment processing by Card type Goal: Parallelized payment processing by Card Type Scenario #1 Luís Marques - @drune - http://lcmarques.com
  21. 21. Consolidate Replay – Workload folding Consolidate Database Replay: Slicing an existing captured workload into subsets by specifying a point in time Allows scale-in testing replaying workload subsets together Our Goal: Parallelized payment processing by Card Type Start of Day Processing VISA Payment Process MasterCard Payment Process OthersCard Payment Process Workload TimeWL Subset #1 WL Subset #2 WL Subset #3 Luís Marques - @drune - http://lcmarques.com
  22. 22. Consolidate Replay – Workload folding Workload from 1 day (7pm to 6am) – capture_payment_day 3 subsets of workload derived from 1 capture: Start of Day Processing VISA Payment Process MasterCard Payment Process OthersCard Payment Process Workload Time Day Finish Replay this! Luís Marques - @drune - http://lcmarques.com
  23. 23. Consolidate Replay – Workload folding BEGIN_TIME and END_TIME are offset in seconds from the start of the capture (tricky) Incomplete calls are discarded to avoid problematic results Example (VISA workload subset #1): • 4500 seconds after start of capture (1 hour and 15 minutes) • Ends after 8100 seconds from the start of capture (1 hour of duration) SQL> EXEC DBMS_WORKLOAD_REPLAY.GENERATE_CAPTURE_SUBSET ( INPUT_CAPTURE_DIR => 'DB_REPLAY_DIR', OUTPUT_CAPTURE_DIR => 'DB_REPLAY_VISA', NEW_CAPTURE_NAME => 'CardVisa_wkld’, BEGIN_TIME => 4500, BEGIN_INCLUDE_INCOMPLETE=> FALSE, END_TIME => 8100, END_INCLUDE_INCOMPLETE=> FALSE, PARALLEL_LEVEL=>1) Luís Marques - @drune - http://lcmarques.com
  24. 24. Consolidate Replay – Workload folding Preprocess capture from all directories: EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_VISA'); EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_MCARD'); EXEC DBMS_WORKLOAD_REPLAY.PROCESS_CAPTURE ('DB_REPLAY_OTHERS'); Create a consolidate replay schedule EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE ('visa_mcard_others_schedule'); SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_VISA') FROM dual; SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_MCARD') FROM dual; SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_OTHERS') FROM dual; EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE; Luís Marques - @drune - http://lcmarques.com
  25. 25. Consolidate Replay – Workload folding Remap connections (DBA_WORKLOAD_CONNECTION_MAP) – You know how to do it  Prepare your consolidation replay: SQL> EXEC DBMS_WORKLOAD_REPLAY.PREPARE_CONSOLIDATED_REPLAY (synchronization=>’OBJECT_ID’, think_time_auto_correct => TRUE ) • Each capture subset have a different SCN. SCN-based sync is not possible • Only OBJECT_ID and OFF are supported as syncronization options • Connect_time_scale and think_time_scale were left at default • Best Results avoiding connect time modifications or think time modifications Luís Marques - @drune - http://lcmarques.com
  26. 26. Consolidate Replay – OBJECT_ID A “more advanced” synchronization scheme is used Ensures that SCN’s for different captures do not clash Better dealing with concurrency when actions do not touch the same object Bugs happen: If you find problems with OBJECT_ID sync you are left to no syncronization Monitor the progress of your replay. A collection of scripts here help - Scripts to Debug Slow Replay (Doc ID 760402.1) Luís Marques - @drune - http://lcmarques.com
  27. 27. Consolidate Replay – Workload folding Start replay clients – At least one per subset wrc system/oracle mode=replay replaydir=/home/oracle/scripts/rat/db_replay_dir/ sleep 1 wrc system/oracle mode=replay replaydir=/home/oracle/scripts/ rat/db_replay_dir/ … Start database replay SQL> EXEC DBMS_WORKLOAD_REPLAY.START_CONSOLIDATED_REPLAY; Luís Marques - @drune - http://lcmarques.com
  28. 28. Review divergences after replay Application Level divergences: • Example: Check if all the VISA cards were processed (manual SQL Script) Data Level divergences: • Different number of rows returned by DML or SQL queries – Don’t match with capture Error Level divergences: • Errors on Capture that are not on replay • Errors on Replay that were not captured • Different errors on capture and replay select DBMS_WORKLOAD_REPLAY.REPORT(replay_id => 1, format => 'HTML') from dual; Luís Marques - @drune - http://lcmarques.com
  29. 29. Sequential vs Parallel: Compare it! Import AWR from previous sequential replay to compare to parallel replay dbms_workload_replay.import_awr(replay_id => 12, staging_schema => 'PAY_A'); Generate a compare replays report spool report_compare_consolidate.html VAR v_clob CLOB BEGIN dbms_workload_replay.compare_period_report (replay_id1 => 11, replay_id2 => 12, format => DBMS_WORKLOAD_REPLAY.TYPE_HTML, result => :v_clob ); END; / PRINT v_clob spool off Luís Marques - @drune - http://lcmarques.com
  30. 30. Sequential vs Parallel: Compare it! Compare DB Time between periods (Sequential TP1 vs Parallel TP2); Easier than read an AWR  Luís Marques - @drune - http://lcmarques.com
  31. 31. Workload: Parallelized payment processing by Card Type Goal: Parallelized payment processing by Card Type and ATM machines payment processing from another database capture Scenario #2 Luís Marques - @drune - http://lcmarques.com
  32. 32. Consolidate Replay – Time shifting Allows shifting a replay to a time in the future and align with other replays Stress testing by align peak loads Peak number of sessions Add another workload here! Luís Marques - @drune - http://lcmarques.com
  33. 33. Consolidate Replay – Time shifting Align “Multibanco ATM processing” and Card Payment Processing to start exactly at same time VISA Payment Process MasterCard Payment Process OthersCard Payment Process Validations Multibanco ATM processing Workload time Workload time Delay time Luís Marques - @drune - http://lcmarques.com
  34. 34. Consolidate Replay – Time shifting Preprocess and delay the workloads to match ATM replay EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE('payment_peak_align'); SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_ATM') FROM dual; SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_VISA', 600) FROM dual; SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_MCARD', 650) FROM dual; SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_OTHERS', 660) FROM dual; EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE; EXEC DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY ('payment_peak_replay', 'payment_peak_align'); Delay secs Luís Marques - @drune - http://lcmarques.com
  35. 35. Workload: Single workload capture on ATM machines payment Goal: Stress test ATM machine payment processing on a very busy day – December 24th Scenario #3 Luís Marques - @drune - http://lcmarques.com
  36. 36. Consolidate Replay – Schema Remapping A single workload consolidated into two different instances of the application Original capture schema: MULTIBANCO_ATM (original) Remapped schema: MULTIBANCO_ATM_B (remapped) Luís Marques - @drune - http://lcmarques.com Validations Multibanco ATM processing Multibanco ATM processing Validations Fees Calc Fees Calc MULTIBANCO_ATM_B Workload time
  37. 37. Consolidate Replay – Schema Remapping Add the same workload twice to a consolidate replay SQL> EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE(’multibanco _stress_schedule'); SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_ATM') FROM dual; SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE ('DB_REPLAY_ATM') FROM dual; EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE; EXEC DBMS_WORKLOAD_REPLAY.INITIALIZE_CONSOLIDATED_REPLAY ('payment_peak_replay', ‘multibanco_stress_schedule'); EXEC DBMS_WORKLOAD_REPLAY.SET_USER_MAPPING (2, ’MULTIBANCO_ATM', ’MULTIBANCO_ATM_B'); Luís Marques - @drune - http://lcmarques.com
  38. 38. Workload: Applied to all previous scenarios Goal: Get a “even” more realist parallelized payment processing Scenario #4 Luís Marques - @drune - http://lcmarques.com
  39. 39. Query Only – Warm up my buffer cache When you start your replay, remember that your cache is "cold” Replay may not be realistic or take too much time – capture is done with already some blocks in buffer cache. Warm-up the buffer cache (if needed) with Query Only Replay before the real workload. Query Only Replay replays read-only statements Luís Marques - @drune - http://lcmarques.com
  40. 40. Query Only – Warm up my buffer cache example EXEC DBMS_WORKLOAD_REPLAY.BEGIN_REPLAY_SCHEDULE ('visa_mcard_others_schedule'); SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_ VISA’, query_only => TRUE) FROM dual; SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_ MCARD’, query_only => TRUE) FROM dual; SELECT DBMS_WORKLOAD_REPLAY.ADD_CAPTURE('DB_REPLAY_ OTHERS’,query_only => TRUE) FROM dual; EXEC DBMS_WORKLOAD_REPLAY.END_REPLAY_SCHEDULE; Luís Marques - @drune - http://lcmarques.com
  41. 41. Final Notes Consolidate Replay can be really helpful on stress testing and scaling up on “brand new” PoC system Comparing periods is important: Capture vs Replay; Consolidate Replay vs Sequential Replays, etc Follow capture and replay recommendations. Make sure that you know your capture and his subsets and start your replay with the “defaults” Try different synchronization options if you find in trouble with OBJECT_ID. Remember that OBJECT_ID and OFF are the only options for Consolidated Replay Luís Marques - @drune - http://lcmarques.com
  42. 42. Q & A Luís Marques - @drune - http://lcmarques.com

×