Migrate 10 TB to Exadata -- Tips and Tricks Presentation at IOUG COLLABORATE 14, April 2014 Las Vegas

1. A M I N A D A T I A
K N O W T E C H S O L U T I O N S I N C .
W O R K I N G W I T H
O R A C L E S I N C E 1 9 8 5
E X A D A T A S I N C E 2 0 1 1
F O C U S
C U S T O M A P P L I C A T I O N D E V E L O P M E N T
Migrate to Exadata X3
Tips and Tricks
Level 3, San Polo 3403, Session 326
Thu, Apr 10, 2014 (12:15 PM - 12:45 PM)

2. Contact Details
 eMail => amin@knowtech.ca
 Twitter => aminadatia
 LinkedIn
 Data Architecture Professionals
 Oracle Exadata (IOUG SIG)
 Oracle Exadata Users
 Oracle User Groups
 IOUG
 ODTUG

3. Agenda
 Background
 Source and Target Environments
 Migration Constraints
 Data Migration Techniques
 Approach
 Non Partitioned Tables
 Partitioned Tables
 Without LOB Columns
 With LOB Columns
 Sub-Partition by Hash Tables
 Oracle Text Indexes without using TTS
 Oracle Label Security without Custom Label Tags

4. Environments
 SOURCE
 Oracle 10.2.0.4
 AIX
 2 Node RAC
 48 CPU per Node
 Target
 Oracle 11.2.0.3
 Exadata X3
 8 Node RAC
 32 CPU per Node
 RAM
 Patchset
 CellPatch

5. Downtime Constraints
 Users mostly query for the past 240 Partitions
 Some queries span 1200 Partitions
 Plan for
 480 Partitions loaded before users allowed access and ETL
loading to resume
 Estimated time was 5 hours and so 6 hours was allocated for
cold migration
 Not much prospect to drop indexes and recreate
 ETL was done for 3 days prior to migration
 Having NO INDEXES resulted in zero ETL
 At the end of the downtime => ETL would start

6. Source AIX Environment
 Tablespaces => 450 (9.8 TB Disk Used)
 Table Partitions distributed across all Tablespaces
 Tables (for Migration)
 Partitioned => 33 (~ 2500 Partitions/Table)
 Non Partitioned => 65 (2 Large Number of Rows)
 Oracle Text Indexes => 4
 Tables with LOBs => 4 (CLOB and BLOB)
 Oracle Label Security => ~ 280 Labels
 Network Link => 1 MBit and 10 MBit

7. Target X3 Environment
 Tablespaces => Same as Source
 Oracle Label Security
 Create Policy
 Create Labels and Tags
 Map New Tags to Old Tags
 Oracle Text
 Define Preferences
 Schema Definitions
 Export from Source with Rows = N
 Import into Target

8. Target X3 Environment
Going from Test (X2) to Production (X3)
 DataPump had a bug => Extremely Slow
 Revert to using 10g EXP
 Virtual Columns Not Recognized
 Schema Definitions (3)
 Export from X2 with (Rows = N)
 Create Oracle Text Preferences
 Create OLS Policy
 Import into X3
 Recreate Objects with Virtual Columns

9. Non Partitioned Tables

10. Non Partitioned Tables … 1
All commands Issued from Exadata
INSERT INTO <Table_Name>
SELECT /*+ PARALLEL (T1,p_Degreee) */
FROM <Source_Table>@DbLink_Pipe T1
PX Elapsed Time
12 57 minutes
24 35 minutes

11. W I T H N O L O B C O L U M N S
W I T H L O B C O L U M N S
S U B - P A R T I T I O N B Y H A S H A N D L O B C O L U M N
Partitioned Tables

12. Partitioned Tables Outcome
Hours
Table

13. W I T H N O L O B C O L U M N S
Partitioned Tables (No LOBs)

14. Partitioned Table (No LOBs) ..1
v_Step := 'Create => '||v_Q_Table||' => '||p_Table_Name;
EXECUTE IMMEDIATE
'create table '||v_Q_Table
||chr(10)||'TABLESPACE '||v_Tablespace_Name
||chr(10)||'NOLOGGING'
||chr(10)||'as select'
||chr(10)||' /*+'
||chr(10)||' parallel (a,'||p_Parallel_Source||')'
||chr(10)||' no_index ('||p_No_Index_Hint||')'
||chr(10)||' */'
||chr(10)||' * from '||p_Table_Name||'@'||p_Data_Source||' a'
||chr(10)||'where a.PartKey
||chr(10)||’ BETWEEN p_Start_Number - '||p_Offset_Start
||chr(10)||' AND p_Start_Number - '||p_Offset_End;

15. Partitioned Table (No LOBs) ..2
v_Step := 'Gather Stats => '||v_Q_Table;
v_Step := ‘Exchange Working Table for Partition';  If RECORDS > 0
Insert Data from Q_Table into Working Table based on Partition_Key.
v_Step := 'Create Matching Partition Indexes';
Create Index on Working Table
Exchange Working Table with Partition

16. Partitioned Table (No LOBs) .. 3
EXECUTE IMMEDIATE
'ALTER TABLE '||p_Table_Name
||chr(10)||' EXCHANGE PARTITION‘
||v_Partition_Name
||chr(10)||' WITH TABLE '
||v_Working_Table
||chr(10)||' INCLUDING INDEXES'
||chr(10)||' WITHOUT VALIDATION'
||chr(10)||' UPDATE GLOBAL INDEXES';

17. Partitioned Table (No LOBs) … 4
Parallel at Source => 2,4,8,12,24
depending on the amount of data
how many jobs were still running
Parallel at Target => usually 8
After 4 hours Resources on AIX were diverted to deal
to Tables with LOB Columns

18. T A B L E S W I T H L O B C O L U M N S
Partition Tables (LOBs)

19. Partitioned Table (LOBs) ..1
 Parallel at Source => 1,2,4,8,16,24,32
 Eventually found out that PX = 1 was best
 By the time we figured out the solution only
one table was left to do
 Submitted 45 jobs on each source node
 using parallel = 1 at source
 Each job took about 15 minutes

20. Partitioned Table (LOBs) ..2

21. Partitioned Table (LOBs) ..3
v_Begin INTEGER := 210;
BEGIN
FOR I in 1..45
LOOP
v_Start := v_Begin + (I - 1);
v_End := v_Start;
v_Job_Name := 'M_B1_'||v_Start||'_'||v_End;
v_Job_Action :=
'BEGIN'
||chr(10)
||'MIGRATION.DIRECT_EXCHANGE(''‘
||p_Partition_LOB_Table
||'''',12,''a'',1,8,'
||v_Start||','||v_End||','''||v_Source||''');'
||chr(10)||'END;';

22. Partitioned Table (LOBs) ..4
BEGIN
DBMS_SCHEDULER.CREATE_JOB
( job_name => v_Job_Name
,job_type => 'PLSQL_BLOCK'
,job_action => v_Job_Action
,start_date => SYSTIMESTAMP
,enabled => TRUE
);
DBMS_SCHEDULER.SET_ATTRIBUTE
( name => v_Job_Name
,attribute => 'instance_stickiness'
,value => FALSE
);
END;
END LOOP;

23. Partition Exchange

24. Partition Exchange .. 2
 Worked fine except for continuous ETL …
 TABLE gets LOCKED for the EXCHANGE
 If the duration was “small” (less than 2 minutes)
then there was not a noticeable impact on the
ETL process
 32 SubPartition Exchange approach did not work
well

25. Migration Summary
Object Type Estimate
Partitions
Completed/Total
Actual
Partitions within
6 Hour Downtime
Actual Time
Non Partitioned Tables (65) 30 minutes 12 minutes
Partitioned Tables
Non LOB Columns (30)
480/2500 26 Tables
completed
1 Table 880
2 Tables 440
1 Table 200
8 Hours
11 Hours
20 Hours
Partitioned Tables
CLOB Column Tables (2 Tables)
1 Table 480/2500
1 Table 40/1800
1000
160
11 Hours
15 Hours
Sub Partitioned (32) Table
BLOB Column
28/1200 16 8 Days!!

26. Sub Partition Exchange

27. Sub Partition Exchange .. 1
 Table with
 Range Partition on two columns
 32 Sub Partitions by Hash on third column
 Getting the Hash Value and finding the subpartition
seemed to slow down the exchange
 Simple insert from Exchange Table also had the
same kind of issue. Worse than Sub Partition
Exchange
 Insert using Sorted by
(ORA_HASH(HashKey,31,0)+1) significantly
improved performance

28. Sub Partition Exchange .. 2
INSERT INTO HASH_LOB_TABLE
SELECT /*+ NO_INDEX(A) */
* FROM Q_TABLE A
ORDER BY
A.PartKey_Keep
,(ORA_HASH(A.<hashkey>,31,0) + 1);

29. Sub Partition Exchange … 3
Time per Record
Not much good for
Simultaneous ETL

30. Sub Partition Exchange … 4
Relative Insert Time/ Record

31. W I T H O U T T R A N S P O R T A B L E T A B L E S P A C E S
Oracle Text Index

32. Oracle Text Environment
 Tables with Text Indexes
 Multi-lingual Documents stored on File System (2 Tables)
 Generated XML from Data (2 Tables)
 Transportable Tablespaces Method for migrating
was estimated to take 20 days
 New data could not be loaded until TTS was
completed

33. Document Indexing
 Using High Degree Parallel (32,64,96)
 One Partition Data could be indexed in about 30 – 60 minutes
 This was less than the time it took to move the files to the new
file system
 Indexing routine checked the status of the file move
 Indexing Started when File move was completed
 For Highlighting Search Items, the Text Query goes
against the Text Document stored on the file system
and so the files needed to be moved

34. Sync Indexing
 Using CTX_USER_PENDING to get the Partition to be
SYNCed did not work well => 18 minutes to return
Partition Names
 Write a View => Less than 1 second
 CTXSYS.DR$PENDING (Parallel 2)
 CTXSYS.DR$INDEX
 CTXSYS.DR$INDEX_PARTITION (Parallel 2)
 Loop through Partitions found “PENDING”
 Job runs every 10 seconds
 Parallel Degree usually
 Text 8 (used 16,24,32,64)
 XML 16 (used 4,8,16,24,32,64,96)

35. Optimize Indexing
 SYNC_INDEX is with Parallel (4,8,16,32,64,96)
 OPTIMIZE_INDEX – Parallel 1
 SYNC_INDEX get Locked out
 CTX_DDL.LOCK_NOWAIT did not resolve the problem
 Optimize done via DBMS_SCHEDULER Job
 Wrote View to exclude Partitions from SYNC_INDEX
which were part of the Optimize Index Job
 Optimize done on Previous Partitions and it takes about
3 Hours to gain 60% - 85% tokens reduction
 Impact on Switch Log Frequency (+ 100/hour)

36. XML Document Indexing
 Using High Degree Parallel (16,32,64)
 One Partition Data could be indexed in about 30 minutes
 This was about the time it took to generate the XML
 XML generation could only be done once the underlying tables
had the data migrated
 Build XML routine checked for all the data tables for
the day had been migrated
 Indexing routine checked the status of the XML
Build
 Indexing Started when XML Build was completed

37. W I T H O U T C U S T O M L A B E L T A G S
Oracle Label Security

38. Setup OLS Metadata
 Create Label Policy
 Create Labels from Source
 Define Mapping between OLD => NEW Tags
 Use CASE/DECODE to convert during Migration of
Table
 WHEN <Label_Tag> = Old Value THEN <New-Label_Tag>
 ELSE <not_matched_tag>
 Issues =>
 Updating Label_Tag after activating OLS on the Table was
painstakingly slow

39. Questions??

40. E N J O Y T H E R E S T O F T H E C O N F E R E N C E
Thank you!!