All Change

5,240 views
5,097 views

Published on

Talk about an Oracle 10.1 Linux to Solaris 10.2 migration

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
5,240
On SlideShare
0
From Embeds
0
Number of Embeds
1,037
Actions
Shares
0
Downloads
51
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • All Change

    1. 1. All Change: The Story of a Migration Dr. Jason Arneil
    2. 2. <ul><li>Introduction </li></ul><ul><li>The Installation </li></ul><ul><li>The Migration </li></ul><ul><li>Conclusion </li></ul>AGENDA
    3. 3. About Me Introduction <ul><li>Jason Arneil </li></ul><ul><li>Oracle DBA/System & Storage Administrator </li></ul><ul><li>Using Oracle since 1998 </li></ul><ul><li>At Nominet since 2001 </li></ul>
    4. 4. About Nominet Introduction <ul><li>Nominet is the internet registry for .uk domain names </li></ul><ul><li>Nominet has just celebrated its 11 th anniversary </li></ul><ul><li>Nominet is run as a not-for-profit company </li></ul><ul><li>Nominet is owned by its members </li></ul><ul><li>There are almost 6 Million .uk domain names </li></ul>
    5. 5. Hardware/Software Stack Introduction Oracle10.1 Veritas Powerpath Solaris 8 EMC
    6. 6. Motivation Introduction <ul><li>Replace ageing SPARC hardware </li></ul><ul><li>New 4-way Sun SPARC servers ~ £50K EACH </li></ul><ul><li>Equivalent 4-way AMD Opteron cost £20K in TOTAL </li></ul><ul><li>Reduce complexity in the software stack </li></ul><ul><li>Ability to store accented characters </li></ul>
    7. 7. Changes Introduction 10.1.0.3 10.2.0.2 Solaris 8 RedHat AS 4 VxFS ASM US7ASCII UTF8 SPARC AMD
    8. 8. Issues <ul><li>This was not just a case of running an upgrade script </li></ul><ul><li>Database was some 100GB in size </li></ul><ul><li>Change in endianess an additional complication </li></ul><ul><li>Downtime allowable - but needed to be the minimum possible </li></ul>Introduction
    9. 9. Initial Server Install <ul><li>Server installation kept to as minimal rpm’s as possible </li></ul><ul><li>Recommended install rpms have many package dependencies </li></ul><ul><li>Both 32-bit and 64-bit versions of certain packages are required </li></ul><ul><ul><li>rpm --query --queryformat “%(NAME)-%(VERSION).%(RELEASE) (%(ARCH)) ” rpm name </li></ul></ul><ul><li>Binutils rpm incorrect with RedHat Update 2 </li></ul><ul><ul><li>rpm -Uvh --force binutils-2.15.92.0.213.0.0.0.2.x86_64.rpm </li></ul></ul>The Installation
    10. 10. ASMLib The Installation <ul><li>Eliminates need to bind raw devices </li></ul><ul><li>Need to install 3 rpms: </li></ul><ul><ul><li>oracleasm-support </li></ul></ul><ul><ul><li>oracleasm-your-kernel </li></ul></ul><ul><ul><li>oracleasmlib </li></ul></ul><ul><li>Must get ASMLib from </li></ul><ul><li>http://www.oracle.com/technology/tech/linux/asmlib/index.html </li></ul>
    11. 11. Other tips The Installation <ul><li>Follow recommendations in the Oracle Validated Linux configurations: </li></ul><ul><li>http://www.oracle.com/technology/tech/linux/validated-configurations/index.html </li></ul><ul><li>Ensure DNS resolving works for hostnames, VIPs </li></ul><ul><li>Use Cluster Verification Utility </li></ul><ul><li>Bond the Interconnect </li></ul>
    12. 12. Hugepages The Installation <ul><li>Translation Lookaside Buffer maps virtual -> physical memory </li></ul><ul><li>Normal memory page size is 4K </li></ul><ul><li>Hugepage size is 2MB resulting in more efficient memory access </li></ul>
    13. 13. <ul><li>Method of moving tablespaces between databases </li></ul><ul><li>Export metadata then copy the actual datafiles </li></ul><ul><li>Plug in the metadata into the target database </li></ul><ul><li>Transporting across platforms new in 10g </li></ul><ul><li>A change of endianess means an rman convert is needed </li></ul>Transportable Tablespaces The Migration
    14. 14. <ul><li>Check your platform is supported for transporting: </li></ul><ul><ul><li>sql> select * from V$TRANSPORTABLE_PLATFORM </li></ul></ul><ul><li>PLATFORM_ID PLATFORM_NAME ENDIAN_FORMAT </li></ul><ul><li>----------- -------------------------------- -------------- </li></ul><ul><li>1 Solaris[tm] OE (32-bit) Big </li></ul><ul><li>2 Solaris[tm] OE (64-bit) Big </li></ul><ul><li>3 HP-UX (64-bit) Big </li></ul><ul><li>4 HP-UX IA (64-bit) Big </li></ul><ul><li>5 HP Tru64 UNIX Little </li></ul><ul><li>6 AIX-Based Systems (64-bit) Big </li></ul><ul><li>7 Microsoft Windows IA (32-bit) Little </li></ul><ul><li>. </li></ul><ul><li>. </li></ul><ul><li>. </li></ul>Transportable Tablespaces The Migration
    15. 15. Transport The Migration <ul><li>Must have a self-contained set of tablespaces to move </li></ul><ul><ul><li>sql> EXECUTE DBMS_TTS.TRANSPORT_SET_CHECK('TBNAME', TRUE); </li></ul></ul><ul><ul><li>sql> select * from TRANSPORT_SET_VIOLATIONS; </li></ul></ul><ul><li>You must make the tablespaces to be moved read only </li></ul><ul><li>Convert the endianess of the datafiles </li></ul><ul><ul><li>rman> convert tablespace TBNAME to platform 'Linux 64-bit for AMD’ format='/home/oracle/%U' ; </li></ul></ul>
    16. 16. Plugging In The Migration <ul><li>Now you need the metadata: </li></ul><ul><ul><li>exp ‘as sys’ transport_tablespace=y file=tbname.dmp tablespaces=tbname </li></ul></ul><ul><li>Plug tablespace in as a “normal” filesystem tablespace: </li></ul><ul><ul><li>imp ‘as sys’ transport_tablespace=y file=‘path to export’ datafiles=‘path to filesystem datafile’ </li></ul></ul><ul><li>Convert from filesystem datafile to ASM datafile: </li></ul><ul><ul><li>rman> convert datafile ‘path to filesystem datafile’ format ‘ASM PATH’ </li></ul></ul>
    17. 17. The Plan The Migration Solaris Linux Rman convert Rman convert Copy datafiles metadata
    18. 18. A problem The Migration <ul><li>Started to notice space calculations incorrect </li></ul><ul><li>Particular problem with 1 critical tablespace </li></ul><ul><li>Assumed we had problems with our created types & custom indexing </li></ul><ul><li>Clearing database down had catastrophic problem: </li></ul><ul><ul><li>sql> drop user TRANSPORTED cascade; </li></ul></ul><ul><li>ERROR at line 1: </li></ul><ul><li>ORA-00600: internal error code, arguments: [ktecgetsh-inc], [2], [], [], [], [], [], [] </li></ul>
    19. 19. Oracle Response The Migration
    20. 20. The cause of the problem The Migration <ul><li>The cause of the instance crashing was the following: </li></ul><ul><ul><li>ORA-00600: internal error code, arguments: [ktecgetsh-inc], [2], [], [], [], [], [], [] </li></ul></ul><ul><ul><li>Non-fatal internal error happened while SMON was doing temporary segment drop. </li></ul></ul><ul><ul><li>Instance terminated by PMON </li></ul></ul><ul><li>Internal table SEG$ info wrong after TTS </li></ul><ul><li>Fixed by patch 5186784 </li></ul>
    21. 21. Data Pump to the Rescue! The Migration <ul><li>Could not wait for a fix to the Transportable Tablespace problems - did not know how long it would take. </li></ul><ul><li>Data Pump new in 10g - export/import on steroids </li></ul><ul><li>Oracle claim Data Pump is 2x faster than export and 15-45 times faster than import </li></ul><ul><li>Restartable with detailed job monitoring </li></ul><ul><li>Automatic tuning and parallel server side execution </li></ul><ul><li>direct import across the network </li></ul>
    22. 22. Data Pump Architecture The Migration Client Process Client Process Status Queue Master table LOG Master Control Process Command and Control Queue Worker Worker Dump file set Data, Metadata & Master Table
    23. 23. Data Pump Basics The Migration <ul><li>Create dumpdir: </li></ul><ul><ul><li>sql> create directory dumpdir as '/home/oracle/dumpdir’ </li></ul></ul><ul><ul><li>sql> grant read, write on directory dumpdir to USER; </li></ul></ul><ul><li>Actually do a Data Pump export: </li></ul><ul><ul><li>expdp user/password schemas=user directory=dumpdir dumpfile=output.dmp </li></ul></ul><ul><li>Do the Data Pump import: </li></ul><ul><ul><li>impdp user/password directory=dumpdir dumpfile=output.dmp </li></ul></ul>
    24. 24. Data Pump Issues The Migration <ul><li>Data Pump expdp did not run trouble free: </li></ul><ul><ul><li>ORA-31693: Table data object ”USER&quot;.&quot;DOMAINS&quot; failed to load/unload and is being skipped due to error: </li></ul></ul><ul><ul><li>ORA-06502: PL/SQL: numeric or value error </li></ul></ul><ul><ul><li>LPX-00210: expected '<' instead of '?' </li></ul></ul><ul><li>Apparently the xml/xsl stylesheets were not loaded into our 10.1 database </li></ul><ul><li>The fix was running $Oracle_Home/rdbms/admin/initmeta.sql </li></ul>
    25. 25. Data Pump Issues The Migration <ul><li>Data Pump impdp also was not trouble free: </li></ul><ul><ul><li>ORA-31693: Table data object ”USER&quot;.&quot;DOMAINS&quot; failed to load/unload and is being skipped due to error: ORA-44002: invalid object name ORA-06512: at &quot;SYS.DBMS_ASSERT&quot;, line 283 ORA-06512: at &quot;SYS.KUPD$DATA&quot;, line 1336 ORA-12801: error signaled in parallel query server P004, instance lr2:NOM2 (2) ORA-29913: error in executing ODCIEXTTABLEOPEN callout ORA-31626: job does not exist ORA-06512: at &quot;SYS.ORACLE_DATAPUMP&quot;, line 19 </li></ul></ul><ul><li>Seems like Data Pump import running in parallel across RAC nodes does not work. Shutdown of 2nd node cleared this problem </li></ul>
    26. 26. Data Pump Issues The Migration <ul><li>We encountered even more problems during an import: </li></ul><ul><ul><li>ORA-02374: conversion error loading table ”USER&quot;.&quot;INVOICES” ORA-12899: value too large for column CONTACT (actual: 162, maximum: 100) ORA-02372: data for row: CONTACT 0X'C3A1C2B3CA98C3C5C28DC2A3C283C2A1C3A3C283C2B3C3' </li></ul></ul><ul><li>This time it was due to changing characterset US7ASCII -> UTF8 </li></ul><ul><li>Fortunately not too many rows affected so data munging was possible </li></ul>
    27. 27. Data Pump Issues The Migration <ul><li>Tried playing with importing directly across network: </li></ul><ul><ul><li>ORA-31693: Table data object &quot;SNOWDROP&quot;.&quot;EMPLOYEE_OLE&quot; failed to load/unload and is being skipped due to error: ORA-06502: PL/SQL: numeric or value error </li></ul></ul><ul><li>Seems like this is either a Bug: </li></ul><ul><ul><li>3369197 - Dpimp : Network Mode Hr.Per_Addresses Ora-00997: Illegal Use Of Long Datatype Errors </li></ul></ul><ul><ul><li>Metalink Note 313496.1 </li></ul></ul><ul><li>Or maybe this is a feature - the above bug & Metalink Note are NOT publicly visible </li></ul>
    28. 28. Final Data Pump Issues The Migration <ul><li>When we came to the day of the migration we hit this problem when exporting: </li></ul><ul><ul><li>expdp system parfile=param.exp </li></ul></ul><ul><li>Export: Release 10.1.0.3.0 - 64bit Production on Sunday, 16 July, 2006 10:10 Copyright (c) 2003, Oracle. ORA-31626: job does not exist ORA-31637: cannot create job SYS_EXPORT_SCHEMA_01 for user SYSTEM ORA-06512: at &quot;SYS.DBMS_SYS_ERROR&quot;, line 95 ORA-06512: at &quot;SYS.KUPV$FT_INT&quot;, line 601 ORA-12850: Could not allocate slaves on all specified instances: 2 needed, 1 allocated </li></ul><ul><li>We managed to export 1 schema without issue but second schema was starting Parallel slaves on both nodes </li></ul>
    29. 29. Data Pump Timings The Migration <ul><li>Use a parameter file: </li></ul><ul><ul><li>dumpfile=user.dmp directory=dumpdir parallel=4 logfile=user.log schemas=user </li></ul></ul><ul><li>Some timings: </li></ul>240 95 13 49 75 55 11 35 network (time mins) Impdp (time mins) Expdp (time mins) size (GB)
    30. 30. ODBC Issue Tips & Conclusions <ul><li>Rule based Optimizer meant to be de-supported in 10g - see Metalink Note: 189702.1 </li></ul><ul><li>ODBC driver was adding /*+ RULE */ hint to certain queries - still performing ok in 10.1.0.3 </li></ul><ul><li>/*+ RULE */ hinted queries in 10.2.0.2 were performing x 100 slower </li></ul><ul><li>This is covered in Metalink Note: 373129.1 </li></ul><ul><li>Added logon trigger for application to ignore optimizer hints _optimizer_ignore_hints </li></ul>
    31. 31. Post-Migration Issues Tips & Conclusions <ul><li>There were quite a few niggling issues post migration </li></ul><ul><li>perl-dbi applications silently dying </li></ul><ul><ul><li>update DBD driver to 1.17 </li></ul></ul><ul><li>Pro*C applications silently dying </li></ul><ul><ul><li>update Oracle Client to 10.2 </li></ul></ul>
    32. 32. Post-Migration Issues Tips & Conclusions <ul><li>Java JDBC problem: </li></ul><ul><ul><li>java.sql.SQLException: OALL8 is in an inconsistent state. </li></ul></ul><ul><li>Character set problem with domain registration system </li></ul><ul><ul><li>ORA-01756: quoted string not properly terminated </li></ul></ul><ul><li>Developers finding counting characters “harder” </li></ul><ul><ul><li>Can no longer guarantee 1 character == 1 byte </li></ul></ul>
    33. 33. Migration Benefits Tips & Conclusions <ul><li>Load on servers is greatly reduced </li></ul><ul><li>Most ran query has gone from 1.6ms to 0.8ms </li></ul><ul><li>RAC related waits decreased </li></ul><ul><li>Stable for 6 months + </li></ul>
    34. 34. Conclusion Tips & Conclusions <ul><li>TEST, TEST, TEST </li></ul><ul><li>Transportable tablespaces is quickest way of moving data </li></ul><ul><ul><li>changing platforms really complicates & slows it down </li></ul></ul><ul><li>Data Pump is much, much faster than old export/import </li></ul><ul><ul><li>Roughly 25GB per hour </li></ul></ul>
    35. 35. Questions? All Change: The Story of a Migration <ul><li>Contact: </li></ul><ul><li>[email_address] </li></ul><ul><li>http://blog.nominet.org.uk </li></ul>

    ×