Migrate 10TB to Exadata Tips and Tricks (Presentation)

483
-1

Published on

Migrate 10 TB to Exadata -- Tips and Tricks
Presentation at IOUG COLLABORATE 14, April 2014
Las Vegas

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
483
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
43
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Migrate 10TB to Exadata Tips and Tricks (Presentation)

  1. 1. A M I N A D A T I A K N O W T E C H S O L U T I O N S I N C . W O R K I N G W I T H O R A C L E S I N C E 1 9 8 5 E X A D A T A S I N C E 2 0 1 1 F O C U S C U S T O M A P P L I C A T I O N D E V E L O P M E N T Migrate to Exadata X3 Tips and Tricks Level 3, San Polo 3403, Session 326 Thu, Apr 10, 2014 (12:15 PM - 12:45 PM)
  2. 2. Contact Details  eMail => amin@knowtech.ca  Twitter => aminadatia  LinkedIn  Data Architecture Professionals  Oracle Exadata (IOUG SIG)  Oracle Exadata Users  Oracle User Groups  IOUG  ODTUG
  3. 3. Agenda  Background  Source and Target Environments  Migration Constraints  Data Migration Techniques  Approach  Non Partitioned Tables  Partitioned Tables  Without LOB Columns  With LOB Columns  Sub-Partition by Hash Tables  Oracle Text Indexes without using TTS  Oracle Label Security without Custom Label Tags
  4. 4. Environments  SOURCE  Oracle 10.2.0.4  AIX  2 Node RAC  48 CPU per Node  Target  Oracle 11.2.0.3  Exadata X3  8 Node RAC  32 CPU per Node  RAM  Patchset  CellPatch
  5. 5. Downtime Constraints  Users mostly query for the past 240 Partitions  Some queries span 1200 Partitions  Plan for  480 Partitions loaded before users allowed access and ETL loading to resume  Estimated time was 5 hours and so 6 hours was allocated for cold migration  Not much prospect to drop indexes and recreate  ETL was done for 3 days prior to migration  Having NO INDEXES resulted in zero ETL  At the end of the downtime => ETL would start
  6. 6. Source AIX Environment  Tablespaces => 450 (9.8 TB Disk Used)  Table Partitions distributed across all Tablespaces  Tables (for Migration)  Partitioned => 33 (~ 2500 Partitions/Table)  Non Partitioned => 65 (2 Large Number of Rows)  Oracle Text Indexes => 4  Tables with LOBs => 4 (CLOB and BLOB)  Oracle Label Security => ~ 280 Labels  Network Link => 1 MBit and 10 MBit
  7. 7. Target X3 Environment  Tablespaces => Same as Source  Oracle Label Security  Create Policy  Create Labels and Tags  Map New Tags to Old Tags  Oracle Text  Define Preferences  Schema Definitions  Export from Source with Rows = N  Import into Target
  8. 8. Target X3 Environment Going from Test (X2) to Production (X3)  DataPump had a bug => Extremely Slow  Revert to using 10g EXP  Virtual Columns Not Recognized  Schema Definitions (3)  Export from X2 with (Rows = N)  Create Oracle Text Preferences  Create OLS Policy  Import into X3  Recreate Objects with Virtual Columns
  9. 9. Non Partitioned Tables
  10. 10. Non Partitioned Tables … 1 All commands Issued from Exadata INSERT INTO <Table_Name> SELECT /*+ PARALLEL (T1,p_Degreee) */ FROM <Source_Table>@DbLink_Pipe T1 PX Elapsed Time 12 57 minutes 24 35 minutes
  11. 11. W I T H N O L O B C O L U M N S W I T H L O B C O L U M N S S U B - P A R T I T I O N B Y H A S H A N D L O B C O L U M N Partitioned Tables
  12. 12. Partitioned Tables Outcome Hours Table
  13. 13. W I T H N O L O B C O L U M N S Partitioned Tables (No LOBs)
  14. 14. Partitioned Table (No LOBs) ..1 v_Step := 'Create => '||v_Q_Table||' => '||p_Table_Name; EXECUTE IMMEDIATE 'create table '||v_Q_Table ||chr(10)||'TABLESPACE '||v_Tablespace_Name ||chr(10)||'NOLOGGING' ||chr(10)||'as select' ||chr(10)||' /*+' ||chr(10)||' parallel (a,'||p_Parallel_Source||')' ||chr(10)||' no_index ('||p_No_Index_Hint||')' ||chr(10)||' */' ||chr(10)||' * from '||p_Table_Name||'@'||p_Data_Source||' a' ||chr(10)||'where a.PartKey ||chr(10)||’ BETWEEN p_Start_Number - '||p_Offset_Start ||chr(10)||' AND p_Start_Number - '||p_Offset_End;
  15. 15. Partitioned Table (No LOBs) ..2 v_Step := 'Gather Stats => '||v_Q_Table; v_Step := ‘Exchange Working Table for Partition';  If RECORDS > 0 Insert Data from Q_Table into Working Table based on Partition_Key. v_Step := 'Create Matching Partition Indexes'; Create Index on Working Table Exchange Working Table with Partition
  16. 16. Partitioned Table (No LOBs) .. 3 EXECUTE IMMEDIATE 'ALTER TABLE '||p_Table_Name ||chr(10)||' EXCHANGE PARTITION‘ ||v_Partition_Name ||chr(10)||' WITH TABLE ' ||v_Working_Table ||chr(10)||' INCLUDING INDEXES' ||chr(10)||' WITHOUT VALIDATION' ||chr(10)||' UPDATE GLOBAL INDEXES';
  17. 17. Partitioned Table (No LOBs) … 4 Parallel at Source => 2,4,8,12,24 depending on the amount of data how many jobs were still running Parallel at Target => usually 8 After 4 hours Resources on AIX were diverted to deal to Tables with LOB Columns
  18. 18. T A B L E S W I T H L O B C O L U M N S Partition Tables (LOBs)
  19. 19. Partitioned Table (LOBs) ..1  Parallel at Source => 1,2,4,8,16,24,32  Eventually found out that PX = 1 was best  By the time we figured out the solution only one table was left to do  Submitted 45 jobs on each source node  using parallel = 1 at source  Each job took about 15 minutes
  20. 20. Partitioned Table (LOBs) ..2
  21. 21. Partitioned Table (LOBs) ..3 v_Begin INTEGER := 210; BEGIN FOR I in 1..45 LOOP v_Start := v_Begin + (I - 1); v_End := v_Start; v_Job_Name := 'M_B1_'||v_Start||'_'||v_End; v_Job_Action := 'BEGIN' ||chr(10) ||'MIGRATION.DIRECT_EXCHANGE(''‘ ||p_Partition_LOB_Table ||'''',12,''a'',1,8,' ||v_Start||','||v_End||','''||v_Source||''');' ||chr(10)||'END;';
  22. 22. Partitioned Table (LOBs) ..4 BEGIN DBMS_SCHEDULER.CREATE_JOB ( job_name => v_Job_Name ,job_type => 'PLSQL_BLOCK' ,job_action => v_Job_Action ,start_date => SYSTIMESTAMP ,enabled => TRUE ); DBMS_SCHEDULER.SET_ATTRIBUTE ( name => v_Job_Name ,attribute => 'instance_stickiness' ,value => FALSE ); END; END LOOP;
  23. 23. Partition Exchange
  24. 24. Partition Exchange .. 2  Worked fine except for continuous ETL …  TABLE gets LOCKED for the EXCHANGE  If the duration was “small” (less than 2 minutes) then there was not a noticeable impact on the ETL process  32 SubPartition Exchange approach did not work well
  25. 25. Migration Summary Object Type Estimate Partitions Completed/Total Actual Partitions within 6 Hour Downtime Actual Time Non Partitioned Tables (65) 30 minutes 12 minutes Partitioned Tables Non LOB Columns (30) 480/2500 26 Tables completed 1 Table 880 2 Tables 440 1 Table 200 8 Hours 11 Hours 20 Hours Partitioned Tables CLOB Column Tables (2 Tables) 1 Table 480/2500 1 Table 40/1800 1000 160 11 Hours 15 Hours Sub Partitioned (32) Table BLOB Column 28/1200 16 8 Days!!
  26. 26. Sub Partition Exchange
  27. 27. Sub Partition Exchange .. 1  Table with  Range Partition on two columns  32 Sub Partitions by Hash on third column  Getting the Hash Value and finding the subpartition seemed to slow down the exchange  Simple insert from Exchange Table also had the same kind of issue. Worse than Sub Partition Exchange  Insert using Sorted by (ORA_HASH(HashKey,31,0)+1) significantly improved performance
  28. 28. Sub Partition Exchange .. 2 INSERT INTO HASH_LOB_TABLE SELECT /*+ NO_INDEX(A) */ * FROM Q_TABLE A ORDER BY A.PartKey_Keep ,(ORA_HASH(A.<hashkey>,31,0) + 1);
  29. 29. Sub Partition Exchange … 3 Time per Record Not much good for Simultaneous ETL
  30. 30. Sub Partition Exchange … 4 Relative Insert Time/ Record
  31. 31. W I T H O U T T R A N S P O R T A B L E T A B L E S P A C E S Oracle Text Index
  32. 32. Oracle Text Environment  Tables with Text Indexes  Multi-lingual Documents stored on File System (2 Tables)  Generated XML from Data (2 Tables)  Transportable Tablespaces Method for migrating was estimated to take 20 days  New data could not be loaded until TTS was completed
  33. 33. Document Indexing  Using High Degree Parallel (32,64,96)  One Partition Data could be indexed in about 30 – 60 minutes  This was less than the time it took to move the files to the new file system  Indexing routine checked the status of the file move  Indexing Started when File move was completed  For Highlighting Search Items, the Text Query goes against the Text Document stored on the file system and so the files needed to be moved
  34. 34. Sync Indexing  Using CTX_USER_PENDING to get the Partition to be SYNCed did not work well => 18 minutes to return Partition Names  Write a View => Less than 1 second  CTXSYS.DR$PENDING (Parallel 2)  CTXSYS.DR$INDEX  CTXSYS.DR$INDEX_PARTITION (Parallel 2)  Loop through Partitions found “PENDING”  Job runs every 10 seconds  Parallel Degree usually  Text 8 (used 16,24,32,64)  XML 16 (used 4,8,16,24,32,64,96)
  35. 35. Optimize Indexing  SYNC_INDEX is with Parallel (4,8,16,32,64,96)  OPTIMIZE_INDEX – Parallel 1  SYNC_INDEX get Locked out  CTX_DDL.LOCK_NOWAIT did not resolve the problem  Optimize done via DBMS_SCHEDULER Job  Wrote View to exclude Partitions from SYNC_INDEX which were part of the Optimize Index Job  Optimize done on Previous Partitions and it takes about 3 Hours to gain 60% - 85% tokens reduction  Impact on Switch Log Frequency (+ 100/hour)
  36. 36. XML Document Indexing  Using High Degree Parallel (16,32,64)  One Partition Data could be indexed in about 30 minutes  This was about the time it took to generate the XML  XML generation could only be done once the underlying tables had the data migrated  Build XML routine checked for all the data tables for the day had been migrated  Indexing routine checked the status of the XML Build  Indexing Started when XML Build was completed
  37. 37. W I T H O U T C U S T O M L A B E L T A G S Oracle Label Security
  38. 38. Setup OLS Metadata  Create Label Policy  Create Labels from Source  Define Mapping between OLD => NEW Tags  Use CASE/DECODE to convert during Migration of Table  WHEN <Label_Tag> = Old Value THEN <New-Label_Tag>  ELSE <not_matched_tag>  Issues =>  Updating Label_Tag after activating OLS on the Table was painstakingly slow
  39. 39. Questions??
  40. 40. E N J O Y T H E R E S T O F T H E C O N F E R E N C E Thank you!!

×