All Change
Upcoming SlideShare
Loading in...5

All Change



Talk about an Oracle 10.1 Linux to Solaris 10.2 migration

Talk about an Oracle 10.1 Linux to Solaris 10.2 migration



Total Views
Views on SlideShare
Embed Views



7 Embeds 747 657 75 6 6 1
url_unknown 1 1



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

All Change All Change Presentation Transcript

  • All Change: The Story of a Migration Dr. Jason Arneil
    • Introduction
    • The Installation
    • The Migration
    • Conclusion
  • About Me Introduction
    • Jason Arneil
    • Oracle DBA/System & Storage Administrator
    • Using Oracle since 1998
    • At Nominet since 2001
  • About Nominet Introduction
    • Nominet is the internet registry for .uk domain names
    • Nominet has just celebrated its 11 th anniversary
    • Nominet is run as a not-for-profit company
    • Nominet is owned by its members
    • There are almost 6 Million .uk domain names
  • Hardware/Software Stack Introduction Oracle10.1 Veritas Powerpath Solaris 8 EMC
  • Motivation Introduction
    • Replace ageing SPARC hardware
    • New 4-way Sun SPARC servers ~ £50K EACH
    • Equivalent 4-way AMD Opteron cost £20K in TOTAL
    • Reduce complexity in the software stack
    • Ability to store accented characters
  • Changes Introduction Solaris 8 RedHat AS 4 VxFS ASM US7ASCII UTF8 SPARC AMD
  • Issues
    • This was not just a case of running an upgrade script
    • Database was some 100GB in size
    • Change in endianess an additional complication
    • Downtime allowable - but needed to be the minimum possible
  • Initial Server Install
    • Server installation kept to as minimal rpm’s as possible
    • Recommended install rpms have many package dependencies
    • Both 32-bit and 64-bit versions of certain packages are required
      • rpm --query --queryformat “%(NAME)-%(VERSION).%(RELEASE) (%(ARCH)) ” rpm name
    • Binutils rpm incorrect with RedHat Update 2
      • rpm -Uvh --force binutils-
    The Installation
  • ASMLib The Installation
    • Eliminates need to bind raw devices
    • Need to install 3 rpms:
      • oracleasm-support
      • oracleasm-your-kernel
      • oracleasmlib
    • Must get ASMLib from
  • Other tips The Installation
    • Follow recommendations in the Oracle Validated Linux configurations:
    • Ensure DNS resolving works for hostnames, VIPs
    • Use Cluster Verification Utility
    • Bond the Interconnect
  • Hugepages The Installation
    • Translation Lookaside Buffer maps virtual -> physical memory
    • Normal memory page size is 4K
    • Hugepage size is 2MB resulting in more efficient memory access
    • Method of moving tablespaces between databases
    • Export metadata then copy the actual datafiles
    • Plug in the metadata into the target database
    • Transporting across platforms new in 10g
    • A change of endianess means an rman convert is needed
    Transportable Tablespaces The Migration
    • Check your platform is supported for transporting:
      • sql> select * from V$TRANSPORTABLE_PLATFORM
    • ----------- -------------------------------- --------------
    • 1 Solaris[tm] OE (32-bit) Big
    • 2 Solaris[tm] OE (64-bit) Big
    • 3 HP-UX (64-bit) Big
    • 4 HP-UX IA (64-bit) Big
    • 5 HP Tru64 UNIX Little
    • 6 AIX-Based Systems (64-bit) Big
    • 7 Microsoft Windows IA (32-bit) Little
    • .
    • .
    • .
    Transportable Tablespaces The Migration
  • Transport The Migration
    • Must have a self-contained set of tablespaces to move
      • sql> select * from TRANSPORT_SET_VIOLATIONS;
    • You must make the tablespaces to be moved read only
    • Convert the endianess of the datafiles
      • rman> convert tablespace TBNAME to platform 'Linux 64-bit for AMD’ format='/home/oracle/%U' ;
  • Plugging In The Migration
    • Now you need the metadata:
      • exp ‘as sys’ transport_tablespace=y file=tbname.dmp tablespaces=tbname
    • Plug tablespace in as a “normal” filesystem tablespace:
      • imp ‘as sys’ transport_tablespace=y file=‘path to export’ datafiles=‘path to filesystem datafile’
    • Convert from filesystem datafile to ASM datafile:
      • rman> convert datafile ‘path to filesystem datafile’ format ‘ASM PATH’
  • The Plan The Migration Solaris Linux Rman convert Rman convert Copy datafiles metadata
  • A problem The Migration
    • Started to notice space calculations incorrect
    • Particular problem with 1 critical tablespace
    • Assumed we had problems with our created types & custom indexing
    • Clearing database down had catastrophic problem:
      • sql> drop user TRANSPORTED cascade;
    • ERROR at line 1:
    • ORA-00600: internal error code, arguments: [ktecgetsh-inc], [2], [], [], [], [], [], []
  • Oracle Response The Migration
  • The cause of the problem The Migration
    • The cause of the instance crashing was the following:
      • ORA-00600: internal error code, arguments: [ktecgetsh-inc], [2], [], [], [], [], [], []
      • Non-fatal internal error happened while SMON was doing temporary segment drop.
      • Instance terminated by PMON
    • Internal table SEG$ info wrong after TTS
    • Fixed by patch 5186784
  • Data Pump to the Rescue! The Migration
    • Could not wait for a fix to the Transportable Tablespace problems - did not know how long it would take.
    • Data Pump new in 10g - export/import on steroids
    • Oracle claim Data Pump is 2x faster than export and 15-45 times faster than import
    • Restartable with detailed job monitoring
    • Automatic tuning and parallel server side execution
    • direct import across the network
  • Data Pump Architecture The Migration Client Process Client Process Status Queue Master table LOG Master Control Process Command and Control Queue Worker Worker Dump file set Data, Metadata & Master Table
  • Data Pump Basics The Migration
    • Create dumpdir:
      • sql> create directory dumpdir as '/home/oracle/dumpdir’
      • sql> grant read, write on directory dumpdir to USER;
    • Actually do a Data Pump export:
      • expdp user/password schemas=user directory=dumpdir dumpfile=output.dmp
    • Do the Data Pump import:
      • impdp user/password directory=dumpdir dumpfile=output.dmp
  • Data Pump Issues The Migration
    • Data Pump expdp did not run trouble free:
      • ORA-31693: Table data object ”USER"."DOMAINS" failed to load/unload and is being skipped due to error:
      • ORA-06502: PL/SQL: numeric or value error
      • LPX-00210: expected '<' instead of '?'
    • Apparently the xml/xsl stylesheets were not loaded into our 10.1 database
    • The fix was running $Oracle_Home/rdbms/admin/initmeta.sql
  • Data Pump Issues The Migration
    • Data Pump impdp also was not trouble free:
      • ORA-31693: Table data object ”USER&quot;.&quot;DOMAINS&quot; failed to load/unload and is being skipped due to error: ORA-44002: invalid object name ORA-06512: at &quot;SYS.DBMS_ASSERT&quot;, line 283 ORA-06512: at &quot;SYS.KUPD$DATA&quot;, line 1336 ORA-12801: error signaled in parallel query server P004, instance lr2:NOM2 (2) ORA-29913: error in executing ODCIEXTTABLEOPEN callout ORA-31626: job does not exist ORA-06512: at &quot;SYS.ORACLE_DATAPUMP&quot;, line 19
    • Seems like Data Pump import running in parallel across RAC nodes does not work. Shutdown of 2nd node cleared this problem
  • Data Pump Issues The Migration
    • We encountered even more problems during an import:
      • ORA-02374: conversion error loading table ”USER&quot;.&quot;INVOICES” ORA-12899: value too large for column CONTACT (actual: 162, maximum: 100) ORA-02372: data for row: CONTACT 0X'C3A1C2B3CA98C3C5C28DC2A3C283C2A1C3A3C283C2B3C3'
    • This time it was due to changing characterset US7ASCII -> UTF8
    • Fortunately not too many rows affected so data munging was possible
  • Data Pump Issues The Migration
    • Tried playing with importing directly across network:
      • ORA-31693: Table data object &quot;SNOWDROP&quot;.&quot;EMPLOYEE_OLE&quot; failed to load/unload and is being skipped due to error: ORA-06502: PL/SQL: numeric or value error
    • Seems like this is either a Bug:
      • 3369197 - Dpimp : Network Mode Hr.Per_Addresses Ora-00997: Illegal Use Of Long Datatype Errors
      • Metalink Note 313496.1
    • Or maybe this is a feature - the above bug & Metalink Note are NOT publicly visible
  • Final Data Pump Issues The Migration
    • When we came to the day of the migration we hit this problem when exporting:
      • expdp system parfile=param.exp
    • Export: Release - 64bit Production on Sunday, 16 July, 2006 10:10 Copyright (c) 2003, Oracle. ORA-31626: job does not exist ORA-31637: cannot create job SYS_EXPORT_SCHEMA_01 for user SYSTEM ORA-06512: at &quot;SYS.DBMS_SYS_ERROR&quot;, line 95 ORA-06512: at &quot;SYS.KUPV$FT_INT&quot;, line 601 ORA-12850: Could not allocate slaves on all specified instances: 2 needed, 1 allocated
    • We managed to export 1 schema without issue but second schema was starting Parallel slaves on both nodes
  • Data Pump Timings The Migration
    • Use a parameter file:
      • dumpfile=user.dmp directory=dumpdir parallel=4 logfile=user.log schemas=user
    • Some timings:
    240 95 13 49 75 55 11 35 network (time mins) Impdp (time mins) Expdp (time mins) size (GB)
  • ODBC Issue Tips & Conclusions
    • Rule based Optimizer meant to be de-supported in 10g - see Metalink Note: 189702.1
    • ODBC driver was adding /*+ RULE */ hint to certain queries - still performing ok in
    • /*+ RULE */ hinted queries in were performing x 100 slower
    • This is covered in Metalink Note: 373129.1
    • Added logon trigger for application to ignore optimizer hints _optimizer_ignore_hints
  • Post-Migration Issues Tips & Conclusions
    • There were quite a few niggling issues post migration
    • perl-dbi applications silently dying
      • update DBD driver to 1.17
    • Pro*C applications silently dying
      • update Oracle Client to 10.2
  • Post-Migration Issues Tips & Conclusions
    • Java JDBC problem:
      • java.sql.SQLException: OALL8 is in an inconsistent state.
    • Character set problem with domain registration system
      • ORA-01756: quoted string not properly terminated
    • Developers finding counting characters “harder”
      • Can no longer guarantee 1 character == 1 byte
  • Migration Benefits Tips & Conclusions
    • Load on servers is greatly reduced
    • Most ran query has gone from 1.6ms to 0.8ms
    • RAC related waits decreased
    • Stable for 6 months +
  • Conclusion Tips & Conclusions
    • Transportable tablespaces is quickest way of moving data
      • changing platforms really complicates & slows it down
    • Data Pump is much, much faster than old export/import
      • Roughly 25GB per hour
  • Questions? All Change: The Story of a Migration
    • Contact:
    • [email_address]