Converting the Oracle Database Character Set

Information Technology Services
DBA-OLTP
CATALINA MARKETING ▪ 200 CARILLON PARKWAY ▪ ST. PETERSBURG FL 33716
Converting the Oracle Database
Character Set
Using CSSCAN and CSALTER Tools
Gus A. Miklos
Senior Database Administrator
Information Technology Services

Converting the Oracle Database Character Set Page 2
Catalina Marketing Corp. June 5, 2015
Table of Contents
Overview.......................................................................................................................................................4
Installing the CSSCAN Tool............................................................................................................................4
Uninstall CSSCAN ......................................................................................................................................5
Verify CSSCAN Installation ........................................................................................................................5
Verify Current Character Set.....................................................................................................................5
Character Set Conversion Process ................................................................................................................6
Process Overview......................................................................................................................................6
Understanding CSSCAN Output ................................................................................................................7
Tablespace Expansion...........................................................................................................................7
Database Scan Summary.......................................................................................................................7
Data Dictionary Conversion Summary..................................................................................................8
Distribution of Convertible, Truncated and Lossy Data by Table .........................................................8
Application Data Details........................................................................................................................9
Pre-CSSCAN Clean Up Process ......................................................................................................................9
Clean Up Invalid Objects...........................................................................................................................9
Drop Orphaned Data Pump Jobs ............................................................................................................11
Purge the DBA Recyclebin.......................................................................................................................11
Remove Sample and/or Demo Schemas and Objects ............................................................................11
Run CSSCAN ................................................................................................................................................12
Clean Up Data Dictionary Objects...............................................................................................................13
SYS.COM$(COMMENT$) .........................................................................................................................13
SYS.HISTGRM$(EPVALUE) .......................................................................................................................14
SYS.REG$(SESSION_KEY) .........................................................................................................................15
SYS.SOURCE$(SOURCE)...........................................................................................................................15
Run CSSCAN for Final Check of Data Dictionary .....................................................................................16
Clean Up Application Data..........................................................................................................................17
Modify Columns That Will Expand on Conversion..................................................................................17
Clean Up Rows/Columns That Contain Lossy Data.................................................................................18
Run Final CSSCAN Check .............................................................................................................................19
Generate List of Tables to be Exported ......................................................................................................20
Export Convertible Table Data....................................................................................................................22

Dealing with XML Datatypes...................................................................................................................22
Run Data Pump Export............................................................................................................................24
Disable Constraints and Triggers ................................................................................................................24
Create Schema List Table........................................................................................................................25
Disable Foreign Key Constraints .............................................................................................................25
Disable Primary (and Unique) Key Constraints.......................................................................................25
Disable Triggers on Exported Tables.......................................................................................................26
Truncate the Data in the Problematic Tables .............................................................................................26
Run CSSCAN for Final Check........................................................................................................................26
Run CSALTER to Convert Character Set ......................................................................................................26
Run Post-Conversion Checks.......................................................................................................................27
Run Post-Conversion Query....................................................................................................................27
Rebuild Catalog and Compile Objects.....................................................................................................28
Perform Data Pump Import ........................................................................................................................29
Errors During Import...............................................................................................................................30
Re-Enable Constraints, Triggers, and Final Clean-Up..................................................................................30
Enable Primary Key Constraints..............................................................................................................30
Enable Foreign Key Constraints ..............................................................................................................30
Enable Triggers........................................................................................................................................31
Reset Run Stats History Retention and Run Database Statistics ............................................................31
Reset Column Comments........................................................................................................................31
Recreate Special Objects.........................................................................................................................32
Drop Intermediate Tables.......................................................................................................................32
Conclusion...................................................................................................................................................32
Appendix A..................................................................................................................................................34

Overview
This document covers the steps required to convert an Oracle database from the Western European (or
any) character set to AL32UTF8. AL32UTF8 is a more universal character set that supports most
languages of the world including, for example, Kanji.
Appendix A lists the pertinent Oracle Support document IDs used to perform all of the conversion steps.
Please refer to this documentation, since various databases will have much different data and,
therefore, will require modifications to the overall conversion process covered in this document.
NOTE: Changing the Oracle character set requires a scheduled outage of the database. The length of the
outage will depend on the amount of data that must be converted. For best results, run CSSCAN with
the database open in RESTRICTED mode. This will produce a more accurate scan as well as ensuring that
other sessions are not modifying data that will then later become a problem during conversion.
In order to run the actual CSALTER procedure, the database must be in RESTRICTED mode and the only
active session must be SYS (as SYSDBA). To place the database in restricted mode, perform a
shutdown/startup. Once the database is back up: ALTER SYSTEM ENABLE RESTRICTED SESSION.
TIP: The entire set of SQL and shell scripts needed to perform all steps for character set migration are
included on the DBA-OLTP SharePoint site in the Oracle -> Scripts section. They are contained in a Unix
TAR archive, so that you can copy and extract them for use on any Oracle Database server. Obviously,
you can modify these scripts as needed, based on the requirements for each database on which
character set conversion is being performed.
Installing the CSSCAN Tool
For Oracle 11g, the CSSCAN tool must be run as SYSDBA using SQL*Plus. The self-contained install script
is in the ORACLE_HOME path: @?/rdbms/admin/csminst.sql
Due to some required modifications, however, there is a version of the install script included in the TAR
file that accompanies this document. The modifications are as follows:
1. Create a dedicated tablespace called CSMIG. It should be approximately 4-8 GB, depending on
the size of the database being converted.
2. The CSMINST script should use the new, dedicated tablespace and not the SYSTEM tablespace:
So after extracting the TAR file to your local directory, simply execute the following to install CSSCAN:
$ sqlplus /nolog
SQL> conn / as sysdba
SQL> set TERMOUT ON
SQL> set ECHO ON
SQL> spool csminst.log
SQL> @csminst.sql

Check the installation log for any errors: > grep –i ora- csminst.log
There should be no errors (of course, you can also view the log to make sure there were no issues).
Uninstall CSSCAN
Uninstalling CSSCAN is a simple two-step process (log-in as SYSDBA):
drop user CSMIG cascade;
drop tablespace CSMIG including contents and datafiles;
Verify CSSCAN Installation
After installing CSSCAN, run a check to make sure the tool is working properly. This is done by running a
scan on one table as shown below (from the OS prompt):
>csscan TABLE='(SYS.SQL_VERSION$)' FROMCHAR=US7ASCII TOCHAR=US7ASCII LOG=dbchk
CAPTURE=N PROCESS=1 ARRAY=1024000
You will be prompted for the SYS password. Below is the entire output:
Character Set Scanner v2.2 : Release 11.2.0.3.0 - Production on Mon Apr 27 15:17:48
2015
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
Username: SYS as sysdba
Password:
Connected to:
Oracle Database 11g Release 11.2.0.3.0 - 64bit Production
Enumerating table to scan...
. process 1 scanning SYS.SQL_VERSION$[AAAAHXAABAAAA1JAAA]
Creating Database Scan Summary Report...
Creating Individual Exception Report...
Scanner terminated successfully.
As long as you see the last highlighted line above, then CSSCAN is properly installed and ready for use.
Verify Current Character Set
Before beginning the process of conversion, verify the current character set for the database by logging
in as SYSDBA and executing the following command:
SQL> SELECT value FROM NLS_DATABASE_PARAMETERS WHERE parameter='NLS_CHARACTERSET';
VALUE
----------------------------------------
WE8MSWIN1252
So for this document, you will convert the database from Western European (WE8MSWIN1252) to
AL32UTF8.

Character Set Conversion Process
Character set conversion is largely dependent on the column and data types found within the database.
Most of the columns (and data) may be converted using the CSALTER tool, but some of the columns and
data will not be convertible and will result in either data loss or truncation (or both). The CSSCAN tool
identifies the schemas and tables that are not directly convertible by listing these as either truncation or
lossy. The problem tables may contain either data dictionary or application data, and both types must
be corrected prior to running the CSALTER tool.
Process Overview
The following is a summary of the steps required to perform character set migration:
1. CSSCAN prerequisite clean up
a. Check for invalid objects and recompile if necessary
b. Drop invalid synonyms or other objects that cannot be compiled and/or are not being
used
c. Remove orphaned Data Pump master tables
d. Check for leftover temporary tables
e. Purge the recyclebin
f. Remove unneeded sample/demo schemas (SCOTT, etc.)
2. Run the CSSCAN tool and identify problematic objects for both the data dictionary and
applications
3. Clean up the data dictionary objects first (fix lossy or truncation issues)
4. Clean up any identified application data that is flagged as lossy or truncation
5. Run CSSCAN again to make sure the data dictionary and application data are 100% convertible
6. Perform a Data Pump data-only export on all application tables flagged as convertible
7. Disable key constraints and any triggers for the problematic tables
8. Truncate the data in the problematic tables
9. Run the CSALTER tool to migrate the character set
10. Run post-conversion checks and steps
11. Perform a Data Pump data-only import from the dump file in Step 7
12. Re-enable referential constraints and triggers, run database statistics, and perform other clean-
up as needed

Understanding CSSCAN Output
CSSCAN will produce three output files. The base file name is specified by the name entered in the LOG
parameter in the CSSCAN command. So in this example, the output files will be:
dbcheck.txt
dbcheck.out
dbcheck.err
The file of primary interest is dbcheck.txt. This file will contain a summary of all of the database objects
that will be convertible, need to be truncated, or experience data loss on conversion. Let’s look at each
section of CSSCAN output.
Tablespace Expansion
The process of character set conversion may increase the column sizes of various tables, so the first
section of the dbcheck.txt file shows how each tablespace may be affected by the column expansion.
Below is a fragment of this section of CSSCAN output.
[Database Size]
Tablespace Used Free Total Expansion
------------------------- --------------- --------------- --------------- ---------------
SYSTEM 654.13M 54.88M 709.00M 2.42M
UNDOTBS_1 9.94M 704.06M 714.00M .00K
TEMP .00K .00K .00K .00K
TOOLS 23.28M 80.72M 104.00M .00K
USERS 7,053.17M 351.91M 7,405.09M 71.36M
AUDIT_TS 1,388.38M 86.63M 1,475.00M 932.00K
.
.
.
CSMIG 22.38M 8,169.63M 8,192.00M .00K
------------------------- --------------- --------------- --------------- ---------------
Total 123,176.91M 20,760.93M 143,937.84M 706.98M
As you can see, the USERS tablespace will grow by 71.36 MB. In this case, there are 351.91 MB of free
space available, so this tablespace will handle the character set migration. There may, however, be
other tablespaces without enough free space to allow for the expansion. In that case, you’ll need to
allocate the additional space needed prior to executing CSALTER.
Database Scan Summary
The next few sections of CSSCAN output will summarize the findings. First, as shown below, it verifies
the parameters being used for CSALTER conversion.
[Database Scan Parameters]
Parameter Value
------------------------------ ------------------------------------------------
CSSCAN Version v2.1
Instance Name dbatest
Database Version 11.2.0.3.0
Scan type Full database

Scan CHAR data? YES
Database character set WE8MSWIN1252
FROMCHAR WE8MSWIN1252
TOCHAR AL32UTF8
Scan NCHAR data? NO
Array fetch buffer size 1024000
Number of processes 4
Capture convertible data? YES
------------------------------ ------------------------------------------------
The Scan Summary below shows the most critical information. In order to run CSALTER for both
dictionary and application objects, CSSCAN must report that all character type data is convertible. As
you can see below, this is not the case, so it will be necessary to perform all of the steps outlined earlier.
[Scan Summary]
Some character type data in the data dictionary are not convertible to the new character set
Some character type application data are not convertible to the new character set
Data Dictionary Conversion Summary
In the next section, CSSCAN shows the percentage of data dictionary data types and their corresponding
state for character conversion. In order for CSALTER to work, there must be zero Truncation or Lossy
columns. At the end of this section, CSSCAN reports this fact.
[Data Dictionary Conversion Summary]
Data Dictionary Tables:
Datatype Changeless Convertible Truncation Lossy
--------------------- ---------------- ---------------- ---------------- ----------------
VARCHAR2 14,886,618 203 12 2
CHAR 17,970,479 0 0 0
LONG 208,837 0 0 0
CLOB 65,980 6,716 0 0
VARRAY 14,151 0 0 0
--------------------- ---------------- ---------------- ---------------- ----------------
Total 33,146,065 6,919 12 2
Total in percentage 99.979% 0.021% 0.000% 0.000%
The data dictionary cannot be safely migrated using the CSALTER script
Distribution of Convertible, Truncated and Lossy Data by Table
The final section of CSSCAN output provides all of the details for convertible, truncated, or lossy columns
by table. Below is a fragment of this output.
[Distribution of Convertible, Truncated and Lossy Data by Table]
USER.TABLE Convertible Truncation Lossy
-------------------------------------------- ---------------- ---------------- ----------------
SYS.COM$ 1 0 0
SYS.EXTERNAL_TAB$ 2 0 0
SYS.HISTGRM$ 76 0 0
SYS.METASTYLESH 179 0 0
SYS.REG$ 0 0 1

SYS.RULE$ 1 0 0
SYSTEM.MASTER_CLU_T_TEST 41 12 0
.
.
.
The first highlighted row above is showing one column in SYS.REG$ that contains lossy data. The second
highlighted row will be truncated upon conversion. Both of these findings must be repaired before
conversion can continue.
TIP: Notice that the second highlighted row appears to be an application test table created in the
SYSTEM schema. This table very likely may be safely dropped. Wherever possible, these types of objects
(e.g., backup, test, or temporary) that CSSCAN is reporting as problematic for conversion may be
dropped. Check with the application team(s) before dropping any objects. (Always ensure you have a
recent database backup before dropping objects and also, just prior to running CSALTER.)
Application Data Details
This section of the output is very similar to the data dictionary section. It shows all of the application
tables that contain problem columns.
Application Data:
USER.TABLE Convertible Truncation Lossy
--------------------------------------------- ---------------- ---------------- ----------------
AS_SCHEDULER.ASAPP_CBD_PARMS 27 0 0
.
.
.
.
MASTER.UPC_DEL_T 0 16 0
MASTER.UPC_MKT_T 6 0 0
MASTER.UPC_T 307 450 0
MASTER.UPC_T_BKP 446 514 0
MASTER.USER_T 22 0 13
Again, this output shows tables with both truncation and lossy issues. Also note, as the tip above
indicated, there is one table that appears to be a backup and very likely may be dropped.
As noted in the Process Overview section above, application tables that show truncation or lossy
columns must be corrected before using Data Pump (data-only) export. In other words, all application
data must be identified as convertible by CSSCAN.
Pre-CSSCAN Clean Up Process
The following sections cover the step-by-step procedures to clean up the source database in preparation
to run CSALTER character set conversion.
Clean Up Invalid Objects
Run chk_invalid_objects.sql and you will get output similar to the following:

Owner Obj Name Obj Type STATUS
-------------------- ------------------------------ ------------------------------ -------
BATCHODM TBLCON SYNONYM INVALID
BATCHODM TBLRESCHAIN SYNONYM INVALID
BATCHODM RESET_SEQUENCES PROCEDURE INVALID
C2C_APP PRICING_PKG_20141125 PACKAGE BODY INVALID
C2C_APP INVOICE_PKG_20140402 PACKAGE BODY INVALID
C2C_BATCH PRICING_PKG SYNONYM INVALID
C2C_BATCH BUS_SEG_CURRENT_V SYNONYM INVALID
.
.
.
PUBLIC PATH SYNONYM INVALID
PUBLIC ABSPATH SYNONYM INVALID
PUBLIC ALL_PATH SYNONYM INVALID
PUBLIC CDS_PKG SYNONYM INVALID
RADAR_APP TBLCON SYNONYM INVALID
ROBER003 TBLRESCHAIN SYNONYM INVALID
ROBER003 BILL_T SYNONYM INVALID
ROBER003 TBLCON SYNONYM INVALID
The first step is to recompile or drop any invalid synonyms. You may use the gen_compile_syn.sql
script, which will attempt to compile both application and PUBLIC synonyms. After running this, you may
still see some that are invalid. This is most likely because the objects referred to have errors, for
example:
-------------------- ------------------------------ ------------------------------ -------
PUBLIC CMS_NDCTGT_V SYNONYM INVALID
PUBLIC CMS_NDCTGT_NDC_V SYNONYM INVALID
DBATEST> select * from CMS_NDCTGT_V;
select * from CMS_NDCTGT_V
*
ERROR at line 1:
ORA-04063: view "MASTER.CMS_NDCTGT_V" has errors
To the extent possible, investigate these objects and see if they can be fixed or dropped (i.e., not in use).
Check with the application team(s) and see if the objects (a view in this case) may be fixed or dropped.
For this document, the synonyms for the invalid views above have been dropped. So now, the
chk_invalid_synonyms.sql script returns no rows.
Finally, if there are some other types of objects that are invalid and can’t be dropped, you may still
proceed with character conversion. In some cases, however, a package or package body may not be
convertible. In that case, you will need to save it off and recreate it after character conversion. This will
be covered in a later section.

Drop Orphaned Data Pump Jobs
Now, run chk_orphaned_dp_jobs.sql. This should return no rows, but if it does, drop the orphaned
tables. The SQL script will show the owner and the table name. The owner is usually (but not always)
SYS.
Purge the DBA Recyclebin
Finally, clear out the DBA_RECYCLEBIN:
DBATEST> select count(*) from dba_recyclebin;
COUNT(*)
----------
76
DBATEST> purge dba_recyclebin;
DBA Recyclebin purged.
DBATEST> select count(*) from dba_recyclebin;
COUNT(*)
----------
0
Remove Sample and/or Demo Schemas and Objects
By default, Oracle installs a set of schemas that may not be in use or not needed depending on your
application requirements. Run the script chk_demo_schemas.sql to see if you have any of these on
the source database.
DBATEST> @chk_demo_schemas.sql
USERNAME
------------------------------
DBSNMP
OUTLN
PERFSTAT
ANONYMOUS
DIP
XDB
Obviously, the schemas listed are valid Oracle-owner schemas and should not be dropped.
The link below is an excellent source that tells you what schemas can be safely dropped.
http://www.orafaq.com/wiki/List_of_default_database_users
If possible, drop or remove any unneeded or unused schemas. You should also check the DBA_REGISTRY
to find out what has been installed:

DBATEST> select COMP_NAME,VERSION,STATUS from dba_registry;
COMP_NAME
--------------------------------------------------------------------------------------
--------------------------------------------
VERSION STATUS
------------------------------ -----------
Oracle XML Database
11.2.0.3.0 VALID
Oracle Database Catalog Views
11.2.0.3.0 VALID
Oracle Database Packages and Types
11.2.0.3.0 VALID
If there are any installed components that are invalid, you should check them, reinstall, or remove them,
again, depending on requirements.
Run CSSCAN
Once CSSCAN has been properly installed and the prerequisites above have been done , the first step is
to perform an initial run on the source database. Depending on the size of the database, CSSCAN may
take a long time to run. So it may be best to run it in the background via a shell script. Below is an
example.
#!/usr/bin/ksh
cd $HOME/gmiklos/csscan/bin
csscan "sys/<password> as sysdba" FULL=Y FROMCHAR=WE8MSWIN1252 TOCHAR=AL32UTF8
ARRAY=1024000 LOG=dbcheck CAPTURE=Y PROCESS=4
print "$ORACLE_SID Scan Complete" | mailx -s "CSSCAN"
gustav.miklos@catalinamarketing.com
exit
TIP: The PROCESSES parameter specifies how many parallel processes are used for the scan. It is CPU
and memory dependent. If you have enough memory and CPUs, you can increase the PROCESSES
parameter to speed up the CSSSAN check.
As explained in an earlier section, once you have the CSSCAN output file, check the tablespace section
and allocate additional space as needed.

Clean Up Data Dictionary Objects
Run chk_data_dict.sql to see a condensed list of problematic data dictionary objects. Sample
output is shown below.
DBATEST> @chk_data_dict.sql
NOTHANDLEDDATADICTCOLUMNS
-------------------------------------------------------------------------------------
SYS.COM$(COMMENT$) - VARCHAR2 - CONVERTIBLE
SYS.HISTGRM$(EPVALUE) - VARCHAR2 - CONVERTIBLE
SYS.REG$(SESSION_KEY) - VARCHAR2 - DATA_LOSS
SYS.SOURCE$(SOURCE) - VARCHAR2 - CONVERTIBLE
SYS.WRI$_OPTSTAT_HISTGRM_HISTORY(EPVALUE) - VARCHAR2 - CONVERTIBLE
SYS.WRI$_OPTSTAT_HISTGRM_HISTORY(EPVALUE) - VARCHAR2 - DATA_LOSS
SYSTEM.MASTER_CLU_T_TEST(AD_MSG_1_TXT) - VARCHAR2 - CONVERTIBLE
SYSTEM.MASTER_CLU_T_TEST(AD_MSG_1_TXT) - VARCHAR2 - EXCEED_SIZE
SYSTEM.MASTER_CLU_T_TEST(AD_MSG_2_TXT) - VARCHAR2 - CONVERTIBLE
SYSTEM.MASTER_CLU_T_TEST(MCLU_DESCR) - VARCHAR2 - EXCEED_SIZE
As you can see, this shows all dictionary objects that will either exceed column size or suffer data loss
(convertible data may be ignored; for 10g and above, the CSALTER procedure will handle these). Objects
that exceed side or show data loss need to be corrected. Note also, as mentioned earlier, that the
highlighted table above appears to be a test table in the SYSTEM schema that can be dropped.
So as a first step, let’s drop this table.
DBATEST> drop table SYSTEM.MASTER_CLU_T_TEST cascade constraints purge;
Table dropped.
NOTE: CSSCAN will scan any objects in the DBA recyclebin. So you must use the purge command when
dropping problematic tables (or other objects).
Now, let’s see how to solve the remaining data dictionary issues. For this, we need to refer to the Oracle
Support documents (specifically, Doc ID 258904.1).
SYS.COM$(COMMENT$)
The first problem table in the CSSCAN list is SYS.COM$. This system table contains column comments on
various tables that are not convertible to the new character set. Run chk_sys_comments.sql to
identify the tables on which column comments will need to be cleared (this query may take a while,
since it must check every column by row id).
DBATEST> @chk_sys_comments.sql
OWNR ONAME
------------------------- ------------------------------------------------------------
MFGAWDS_APP INCENTIVE_BLIND_T

Obviously, you will want to save off the comments and add them back to the table after migration. You
can do this using the gen_save_sys_col_comments.sql script. You must pass in the schema owner and
table name. This will generate a SQL script that will reset the comments after conversion. It appends the
name of the table to the generated SQL script. So, for example:
DBATEST> @gen_save_sys_col_comments.sql MFGAWDS_APP INCENTIVE_BLIND_T
This will produce a new script called add_comments-INCENTIVE_BLIND_T.sql that you can use after
conversion to add the column comments back to the table.
Now that the column comments for this table are saved off, clear the column comments using the
gen_clear_sys_col_comments.sql script.
DBATEST>@gen_clear_sys_col_comments.sql MFGAWDS_APP INCENTIVE_BLIND_T
Do this for all of the tables and columns listed in the initial check.
SYS.HISTGRM$(EPVALUE)
For issues with this dictionary item, perform the following:
 Check the current retention of statistics
 The default value is 31
select systimestamp - dbms_stats.get_stats_history_availability from dual;
 Now disable statistics retention
exec dbms_stats.alter_stats_history_retention(0);
 Remove all statistics
exec DBMS_STATS.PURGE_STATS(systimestamp);
 Check there are no more left
select count(*) from SYS.WRI$_OPTSTAT_HISTGRM_HISTORY;
 This should clear SYS.WRI$_OPTSTAT_HISTGRM_HISTORY(EPVALUE)
 After the character set change you will set the retention back to the original value
exec dbms_stats.alter_stats_history_retention(31);
Now check if there are any table histogram statistics that are not convertible by running
analyze_histgrm.sql. If this returns rows for either CONVERTIBLE DATA or EXCEPTIONAL DATA, then
you must delete the histogram statistics for each table. Use the gen_del_hist_stats.sql script to
generate a dynamic script that will delete the statistics for these problem tables
(run_clr_tbl_histgrm.sql).
Re-run the analyze script and it should now return no rows.
DBATEST> @analyze_histgrm.sql
Table: SYS.HISTGRM$
Error: CONVERTIBLE DATA

no rows selected
Table: SYS.HISTGRM$
Error: EXCEPTIONAL DATA
no rows selected
NOTE: After running CSALTER to migrate to the new character set, you must gather database statistics:
EXEC DBMS_STATS.GATHER_DATABASE_STATS(DEGREE=> 'x')
-- where x is the number of CPU cores.
Also, be sure to reset the stats history retention to the original value (31).
SYS.REG$(SESSION_KEY)
If this item shows up in the CSSAN report, it is most likely due to a session being reported other than
SYSDBA. As mentioned previously, CSSCAN should be run in restricted mode (as SYS only). You can check
the type of connection that was registered by running the chk_sys_reg.sql query. For example:
DBATEST> @chk_sys_reg.sql
SUBSCRIPTION_NAME LOCATION_NAME usr# namespace
-------------------- ------------------------------ ----- ---------
REG_TIME
---------------------------------------------------------------------------
"SYS"."ALERT_QUE":"N net8://(ADDRESS=(PROTOCOL=tcp) 19 1
GTESTCM4_CA_3872_ITO (HOST=10.22.3.139)(PORT=47632)
CUSQA3" )?PR=0
24-APR-15 10.58.40.556612 AM -04:00
In this case, there is only one row and, by querying DBA_USERS on user_id=19, it is an old entry (see the
timestamp) from a connection by the OEM agent:
DBATEST> select username from dba_users where user_id=19;
USERNAME
------------------------------
DBSNMP
NOTE: If the OEM agent is running on the server on which the target database resides, then if possible,
stop the agent until character set migration is complete. If you cannot shut down the OEM agent, then
you must remove the database from the OEM targets until character set conversion has been
successfully completed.
In our example, since the database was shut down and then restarted in restricted mode, this table can
be safely truncated as this connection is no longer valid.
SYS.SOURCE$(SOURCE)
This item is referencing an object (usually a package, package, body, procedure, etc.) that cannot be
converted to the new character set (usually attributed to the fact that there are literals being used in the

object that are not covertible). So first, find the row id in the dbcheck.err file for the object. Use the vi
(or other) Unix editor and search for SOURCE$. You should see a section similar to the following:
User : SYS
Table : SOURCE$
Column: SOURCE
Type : VARCHAR2(4000)
Number of Exceptions : 0
Max Post Conversion Data Size: 4000
ROWID Exception Type Size Cell Data(first 30 bytes)
------------------ ------------------ ----- ------------------------------
AAAABHAABAAAHh+ABn convertible 'Expédition des
AAAABHAABAAAHi3AA/ convertible || 'Le numéro d
AAAABHAABAAAHi3AA2 convertible || 'Nous avons
Now run the chk_src_rowid.sql query passing in the first row id and this will give the owner, object
name and object type:
DBATEST> @chk_src_rowid.sql AAAABHAABAAAHh+ABn
75452
OWNR ONAME OTYPE STATUS
-------------------- ------------------------------ -------------------- -------
CIMS_APP BATCH_FILE_PKG PACKAGE BODY VALID
As you can see, this is a package body in the CIMS_APP schema. Check with the applications team(s) to
find out if this package is in use or if it can be dropped. Alternatively, use TOAD or SQL*Developer to
save off the source script, so you can recreate the object after character set conversion.
The final step is to check for data dictionary (source) corruption. Run the
chk_orphaned_obj_source.sql script. This should return no rows. If this query returns any rows then
there are entries in SOURCE$ that are not mapped to an existing object. In other words, the obj# in
SYS.SOURCE$ contains object id's that do not exist in DBA_OBJECTS. Double check that the obj# is really
not known in DBA_OBJECTS and log an Oracle SR for the database "Dictionary Corruption" team to solve
the orphaned rows.
Run CSSCAN for Final Check of Data Dictionary
Once the above steps have been completed, run another scan to check dictionary objects. If the above
steps have been successful, you should then see the following in the CSSCAN output:
[Scan Summary]
All character type data in the data dictionary are convertible to the new character set
Some character type application data are not convertible to the new character set

--------------------- ---------------- ---------------- ---------------- ----------------
VARCHAR2 2,899,270 0 0 0
CHAR 158,624 0 0 0
LONG 208,147 0 0 0
CLOB 404 443 0 0
VARRAY 14,151 0 0 0
--------------------- ---------------- ---------------- ---------------- ----------------
Total 3,280,596 443 0 0
The data dictionary can be safely migrated using the CSALTER script
NOTE: For Oracle 10g and up no further action is required on the data dictionary for convertible CLOB
data. The CSALTER procedure will successfully convert CLOB columns to the new character set.
So now that the data dictionary is ready for conversion, you may proceed to prepare application data for
conversion.
Clean Up Application Data
The primary goal of this step is to reduce the number of tables that will need to be exported and
imported for character set conversion.
Modify Columns That Will Expand on Conversion
The first step is to identify and modify table columns that will exceed their size when the character set is
converted. Once you identify the problematic columns, you can modify them and thus eliminate the
need to export the owning table.
Run the chk_column_expansion.sql script to identify the columns that need expansion. A portion of
the output is shown below.
DBATEST> @chk_column_expansion.sql
output is:
owner.table_name column_name - column_type - max expansion in Bytes.
....................................................................
CHKDIR.MTRX_BRND_T (BRND_DESCR) - VARCHAR2 - 51 Bytes .
CHKDIR.MTRX_T (CAT_SEG_DESCR) - VARCHAR2 - 34 Bytes .
CHKDIR.MTRX_T (CLIENT_NM) - VARCHAR2 - 32 Bytes .
CHKDIR.MTRX_T (MTRX_DESCR) - VARCHAR2 - 68 Bytes .
CHKDIR.MTRX_T (SPCL_INSTRUCT_DESCR) - VARCHAR2 - 66 Bytes .
CMCPORTAL.RETAILPROGRAMENTRYDATA_T (APPENDED_DESCR) - VARCHAR2 - 22 Bytes .
CRMS.TBLCALL (NOTES) - VARCHAR2 - 257 Bytes .
CRMS.TBLCOUP (DESCR) - VARCHAR2 - 32 Bytes .
CRMS.TBLERROR (ERROR) - VARCHAR2 - 257 Bytes .
So if you check the current column size for the CAT_SEG_DESCR column of CHKDIR.MTRX_T, you see
that it is currently 30 bytes. And CSSCAN is showing that this needs to be expanded to 34 bytes.

So now, for all the columns listed in the above output, you need to modify each column to the
appropriate size.
You can use the gen_modify_column_size.sql script to create dynamic SQL that will perform the
required column modifications.
Clean Up Rows/Columns That Contain Lossy Data
Now you must identify rows (by rowid) that will suffer data loss on conversion. The rows are identified
in the dbcheck.err output file produced by CSSCAN. This is usually a large file, since it also includes all
convertible data. To first get an idea of how many rows are lossy, perform a grep operation as follows:
(dbatest)> grep -i "lossy conversion" dbcheck.err
AAAJ3SABsAAAAL/AAx lossy conversion 93
AAAJ3SABsAAAAL/ABo lossy conversion ( (ÈU
AAAJ3SABsAAAAMBABv lossy conversion "!AAJ3SABsAAAAMAABK lossy conversion 39 /ßRå
AAAJ3SABsAAAAMCABa lossy conversion 3#s conversion &.
AAAJ3SABsAAAAMEABP lossy conversion 93 0014
AAAJ3SABsAAAAL/AAf lossy conversion "3#Ê
AAAJ3SABsAAAAL/ABo lossy conversion (
AAAJ3SABsAAAAMAABK lossy conversion =PV
AAAJ3SABsAAAAMCAA0 lossy conversion !
In this case, there are not a lot of rows involved, so it may be possible to convert the problem columns in
these rows to the new character set. Use the gen_convert_lossy_rows.sql script to convert the
problem columns. A sample of the run script is shown here:
update "MASTER".USER_T set PREV_PSWD_TXT = (select convert
(PREV_PSWD_TXT,'WE8MSWIN1252','AL32UTF8') from "MASTER".USER_T where rowid =
'AAAJ3SABsAAAAL/AAf') where rowid ='AAAJ3SABsAAAAL/AAf';
This should clean up the majority of lossy rows. In some cases, however, the data in a particular row
may not be convertible to the new character set:
DBATEST> update "MASTER".USER_T set CURR_PSWD_TXT = (select convert
(CURR_PSWD_TXT,'WE8MSWIN1252','AL32UTF8') from "MASTER".USER_T where rowid =
'AAAJ3SABsAAAAMCAAJ') where rowid ='AAAJ3SABsAAAAMCAAJ';
update "MASTER".USER_T set CURR_PSWD_TXT = (select convert
(CURR_PSWD_TXT,'WE8MSWIN1252','AL32UTF8') from "MASTER".USER_T where rowid =
'AAAJ3SABsAAAAMCAAJ') where rowid ='AAAJ3SABsAAAAMCAAJ'
*
ERROR at line 1:
ORA-12703: this character set conversion is not supported
There are not many options at this point. The first is to check the data and see if the row can be deleted
or the lossy column modified. In this particular case, we see the following data for this rowid:
DBATEST> select * from master.user_t where rowid='AAAJ3SABsAAAAMCAAJ';
USER_CD USER_LAST_NM USER_FIRST_NM A
------------------------------ ------------------------------ ------------------------------ -

EMAIL_ADDR COUNTRY_NBR CURR_PSWD_TXT
PREV_PSWD_TXT PSWD_CHG_
------------------------------------------------------------ ----------- -------------------- ---
----------------- ---------
LOCALE_ID_NBR
-------------
chasek Kearl Chase N
Chase Kearl 1 &?.?/?R?
"?/?>EF 26-FEB-01
1033
In this case, it appears that this is a password field. But note that it has not changed since Feb 2001. So it
may be possible to simply delete the row or update the column. In either case, this data will be lost on
conversion, so you may need to investigate further and engage application team(s) to make an informed
decision on next steps. For the purposes of this document (and since this is a test database), we have set
the column to a NULL value.
Run Final CSSCAN Check
Once all of the lossy application data has been dealt with, run CSSCAN again to be sure that the
database is now 100% convertible to the new character set. (It may be necessary to re-run some of the
previous steps if you still see lossy or truncation issues—primarily, this is usually due to not finding all
columns that may need to be expanded. )
The final CSSCAN run is now showing exactly what we need to see in order to run the CSALTER
procedure.
[Database Scan Parameters]
Parameter Value
------------------------------ ------------------------------------------------
CSSCAN Version v2.1
Instance Name dbatest
Database Version 11.2.0.3.0
Scan type Full database
Scan CHAR data? YES
Database character set WE8MSWIN1252
FROMCHAR WE8MSWIN1252
TOCHAR AL32UTF8
Scan NCHAR data? NO
Array fetch buffer size 1024000
Number of processes 8
Capture convertible data? YES
------------------------------ ------------------------------------------------
[Scan Summary]
All character type data in the data dictionary are convertible to the new character set
All character type application data are convertible to the new character set

--------------------- ---------------- ---------------- ---------------- ----------------
VARCHAR2 2,897,421 0 0 0
CHAR 158,712 0 0 0
LONG 208,112 0 0 0
CLOB 720 816 0 0
VARRAY 14,151 0 0 0
--------------------- ---------------- ---------------- ---------------- ----------------
Total 3,279,116 816 0 0
The data dictionary can be safely migrated using the CSALTER script
As the CSSCAN report shows, all remaining data dictionary and application data can be safely migrated
using CSALTER. Nevertheless, it is still necessary to export all of the data that is convertible and then
import it back into the database once the character set migration is completed. As stated in the Oracle
documentation:
It is MANDATORY to export and truncate/delete ALL "Convertible" User / Application Data.
Generate List of Tables to be Exported
In order to get the list of tables that need to be exported, execute the create_export_tbl.sql script.
This will create an export_list table in the SYS schema that you will use for other steps in this
document. Now, generate a list of all tables that need to be exported (export_tbl.lst) in a format that
you can use in the export dump parfile. Execute gen_export_tbl_list.sql to produce the table list.
Cut and paste this output into the parfile. Below is a fragment of the final parfile.
USERID='/ as sysdba'
TABLES=(AS_SCHEDULER.ASAPP_CBD_PARMS,
AS_SCHEDULER.GJF_CHERYL1,
AS_SCHEDULER.NTR_IMPORT_MFG_EMAILS,
AS_SCHEDULER.PMDB_PROJECTS,
AS_SCHEDULER.PMDB_PROJECTS_0917,
AS_SCHEDULER.PRD_PROGRAMMCLUS,
AS_SCHEDULER.PRD_PROGRAMMCLUS_PREFIX,
AS_SCHEDULER.PRD_PROGRAMMCLUS_PREFIXM,
AS_SCHEDULER.PRD_PROGRAMNOTES,
.
.
.
RADAR.MS_RPT_OFFER_MSG_T,
RADAR.RADAR_RPT_NM_T,
RTLDIR.MATRIX_BRAND_T,
RTLDIR.MATRIX_T)
EXCLUDE=STATISTICS
CONTENT=DATA_ONLY
DIRECTORY=CSSCAN_DIR
DUMPFILE=expdp_csscan_tbls.dmp
JOB_NAME=expdp_csscan
LOGFILE=expdp_csscan_tbls.log

Closely examine the export table list for tables that can be dropped. Also, note that the export table list
is generated by querying the CSMIG.CSMV$ERRORS table. Below are some examples of tables that
probably can be dropped (and thus, do not need to be exported).
AS_SCHEDULER.PRD_PROGRAMS_BKP050914
C2C_APP.CON_HIST_T_BAK
MASTER.MEDIA_MSG_AUDIT_T,
MASTER.MEDIA_MSG_AUDIT_T_b35b6adc
The first two tables above are obviously backups. You may want to check with the application team(s) to
see if these can be dropped. If so, there is obviously no need to export them. The highlighted table is
also suspect due to the unusual naming convention. Using SQL*Plus, first check the row count. But note
that depending on how the table was created you may get something like this:
DBATEST> select count(*) from MASTER.MEDIA_MSG_AUDIT_T_b35b6adc;
select count(*) from MASTER.MEDIA_MSG_AUDIT_T_b35b6adc
*
ERROR at line 1:
ORA-00942: table or view does not exist
If you get this error, it’s mostly likely because the table was created using double-quotes and the
DBA_TABLES view won’t find it. (And in fact, Data Pump export will not find it either, so you won’t be
able to export the data). If you specify the owner and table name in double quotes, then it works.
DBATEST> select count(*) from "MASTER"."MEDIA_MSG_AUDIT_T_b35b6adc";
COUNT(*)
----------
104
Finally, check the creation date and/or the last time any DDL changes were made. The DBA_OBJECTS
view will be able to find the object without using double-quotes.
DBATEST> select created, last_ddl_time from dba_objects where owner='MASTER' and
object_name='MEDIA_MSG_AUDIT_T_b35b6adc';
CREATED LAST_DDL_
--------- ---------
23-AUG-11 23-AUG-11
As you can see, this table was created in 2011, so it’s very likely a candidate to be dropped. If, however,
the application team(s) say that the table is in use, then the only alternative is either to rename the
table or use CTAS to recreate the table with proper naming, so that you can export the data. If the table
needs to be renamed or recreated, keep in mind the following:
 Oracle Database automatically transfers integrity constraints, indexes, and grants on the old
object to the new object.

 Oracle Database invalidates all objects that depend on the renamed object, such as views,
synonyms, and stored procedures and functions that refer to a renamed table.
So use caution with this approach. Once the table is correctly renamed, include it in the parfile for Data
Pump export. Likewise, if the table was dropped, remove it from the parfile.
Export Convertible Table Data
It is now time to export the data that CSSCAN is flagging as convertible during character set conversion.
There are, however, some issues that must be handled before running the export.
Dealing with XML Datatypes
Oracle 11g has a known bug for tables containing XML datatypes (and in conjunction with character set
conversion). The bug will not appear during the export process, but it may cause the following error
when using Data Pump Import:
ORA-31693: Table data object "MASTER"."MCLU_XML_T" failed to load/unload and is being
skipped due to error:
ORA-29913: error in executing ODCIEXTTABLEFETCH callout
ORA-00600: internal error code, arguments: [klxpr_73], [3], [0], [1], [10025], [101],
[], [], [], [], [], []
NOTE: Bug 8528039 - IMPDP errors / conversion problems importing XMLType data with NLS conversion
(Doc ID 8528039.8)
The work-around for this bug is to use the old (9i) exp/imp commands. So first, let’s find the tables from
our export list that contain XML datatype columns. Run the chk_xml_columns.sql script to identify
tables with XML columns
OWNER TABLE_NAME COLUMN_NAME
------------------------------ ------------------------------ ------------------------------
MASTER IMAGE_T IMAGE_XML
MASTER AWARD_XML_T XML_DATA
MASTER OFFER_XML_T XML_DATA
MASTER MCLU_XML_T XML_DATA
MASTER MCLU_LAYOUT_XML_T MCLU_XML
MFGAWDS_APP GPS_BUS_RULES_JOB_T JOB_XML
CDS CDS_JOB_T OUTPUT_XSD
CDS CDS_JOB_T INPUT_XSD
CDS CDS_JOB_REQ_T PARMS_XML
Now run gen_xml_exp_list.sql and this will give you the list of tables (xml_export_tbl.lst) with
XML data types that you can cut and paste into the parfile. You will need to export these tables using
the original exp function using a parfile with the “.dat” extension as follows:
USERID="/ as sysdba"
TABLES=(MASTER.IMAGE_T,
MASTER.AWARD_XML_T,

MASTER.OFFER_XML_T,
MASTER.MCLU_XML_T,
MASTER.MCLU_LAYOUT_XML_T,
MFGAWDS_APP.GPS_BUS_RULES_JOB_T,
CDS.CDS_JOB_T, CDS.CDS_JOB_REQ_T)
ROWS=Y
GRANTS=N
INDEXES=N
CONSTRAINTS=N
STATISTICS=NONE
FILE=exp_xml_tbls.dmp
LOG=exp_xml_tbls.log
Notice that since we’re only interested in the data, we have not exported grants, indexes, constraints, or
statistics. Below is the sample output (you should get no errors).
About to export specified tables via Conventional Path ...
Current user changed to MASTER
. . exporting table IMAGE_T 22835 rows exported
. . exporting table AWARD_XML_T 2860 rows exported
. . exporting table OFFER_XML_T 305 rows exported
. . exporting table MCLU_XML_T 0 rows exported
. . exporting table MCLU_LAYOUT_XML_T 576 rows exported
Current user changed to MFGAWDS_APP
. . exporting table GPS_BUS_RULES_JOB_T 2599 rows exported
Current user changed to CDS
. . exporting table CDS_JOB_T 12 rows exported
. . exporting table CDS_JOB_REQ_T 3872 rows exported
Export terminated successfully without warnings.
After character set migration, you should be able to import the tables using the imp function, again,
using a parfile as follows:
USERID="/ as sysdba"
FROMUSER=MASTER
TOUSER=MASTER
TABLES=(IMAGE_T,
AWARD_XML_T,
OFFER_XML_T,
MCLU_XML_T,
MCLU_LAYOUT_XML_T)
FROMUSER=MFGAWDS_APP
TOUSER=MFGAWDS_APP
TABLES=(GPS_BUS_RULES_JOB_T)
FROMUSER=CDS
TOUSER=CDS
TABLES=(CDS_JOB_T, CDS_JOB_REQ_T)
DATA_ONLY=Y
FILE=exp_xml_tbls.dmp
LOG=imp_xml_tbls.log
And again, the import should show no errors:

Export file created by EXPORT:V11.02.00 via conventional path
import done in US7ASCII character set and AL16UTF16 NCHAR character set
import server uses AL32UTF8 character set (possible charset conversion)
. importing MASTER's objects into MASTER
. . importing table "IMAGE_T" 22835 rows imported
. . importing table "AWARD_XML_T" 2860 rows imported
. . importing table "OFFER_XML_T" 305 rows imported
. . importing table "MCLU_XML_T" 0 rows imported
. . importing table "MCLU_LAYOUT_XML_T" 576 rows imported
. importing MFGAWDS_APP's objects into MFGAWDS_APP
. . importing table "GPS_BUS_RULES_JOB_T" 2599 rows imported
. importing CDS's objects into CDS
. . importing table "CDS_JOB_T" 12 rows imported
. . importing table "CDS_JOB_REQ_T" 3872 rows imported
Import terminated successfully without warnings.
TIP: The gen_export_tbl_list.sql script will not include tables with XML columns, so can use the
export_tbl.lst directly in the Data Pump parfiles (for export and import steps).
Run Data Pump Export
Find a landing zone on the server for the export dump file(s) and create a directory (if necessary). For
example,
DBATEST> create directory CSSCAN_DIR as ‘/testcm_export/csscan’;
DBATEST> grant read, write on directory CSSCAN_DIR to SYS;
Since you don’t know the exact size of the export, nor how long it will take, you can use a shell script to
run the export in the background. So from the Unix prompt:
>nohup expdp_csscan.sh &
Once the export is completed, check the log for any errors.
TIP: For all succeeding steps in this document, be sure to check the output logs. Each of the scripts
provided are spooled off. So for a quick check you can execute the following Unix command:
>grep –i ora- <filename.log>
And, of course, you can visually check the logs for errors.
Disable Constraints and Triggers
Strictly speaking, only constraints on the problematic tables (and triggers) need to be disabled. But
finding all of the referential dependencies may be time-consuming and cumbersome. The goal here is to
be able to truncate (and later, import) the data into the problematic tables (without running into
integrity errors). For this process, a broad-brush approach is used.

Create Schema List Table
Most of the remaining scripts used to prepare the database for character set conversion rely on a
generated table created in the SYS schema that contains all of the application owners that contain table
objects. As SYSDBA, run the create_schema_list.sql script. The main body of this script is as follows:
drop table schema_list purge;
create table schema_list as
select distinct owner from dba_objects
where object_type='TABLE'
order by 1
/
This creates an ordered table (in SYS) of all schemas that own table objects. Obviously, you must now
delete the Oracle-owned schemas from the schema_list table. You want to narrow this down only to
application owners that contain tables. So you must execute a step similar to what is shown below to
delete owners/schemas that are not relevant to the character set migration.
DBATEST> delete from schema_list where owner in (
2 'APPQOSSYS',
3 'CSMIG',
4 'DBSNMP',
5 'OUTLN',
6 'PERFSTAT',
7 'SYS',
8 'SYSTEM',
9 'XDB');
8 rows deleted.
DBATEST> commit;
Commit complete.
Disable Foreign Key Constraints
Before you can truncate the tables in the export list, you must disable all referential constraints.
Execute the gen_disable_global_fk.sql script to disable all foreign keys based on the schema list
generated above. By using this broad approach (disabling all constraints for all table object owners), you
will eliminate errors like the following in the next step:
ORA-02297: cannot disable constraint (C2C_APP.CON_T_PK) - dependencies exist
Disable Primary (and Unique) Key Constraints
After you have disabled the FK constraints, run the gen_disable_global_pk.sql script to do the same
with the primary (and unique) key constraints. As usual, check the logs for any errors and deal with them
as appropriate. In the process of documenting character set conversion, each of these scripts have been
tested and by using this database-wide approach, the character set migration went smoothly.

Disable Triggers on Exported Tables
In this case, it is not necessary to disable all triggers in the database (as it was with the constraints). You
only need to disable the triggers that are related to the owners/tables from the export list table you
created in Step 6. Run the gen_disable_trig.sql script to disable these triggers.
Truncate the Data in the Problematic Tables
Run the gen_trunc_tbls.sql to truncate the data in the tables that were marked as convertible by
CSSCAN and that you exported in Step 6. Again, be sure to check the log file for any errors or issues.
Run CSSCAN for Final Check
You are now ready to run CSSCAN on final time using the steps outlined in a previous section. Once the
CSSCAN tool has finished, run the csscan_final_chk.sql script. This should now return no rows.
DBATEST> @csscan_final_chk.sql
no rows selected
The database is now ready for character set conversion.
Run CSALTER to Convert Character Set
The CSALTER procedure is located in the ORACLE_HOME path at:
@?/rdbms/admin/csalter.plb
The procedure is self-contained, so you can copy it to the directory of your choice (it is included in the
TAR file that comes with this document). To run CSALTER, do the following:
1. SHUTDOWN IMMEDIATE
2. STARTUP
3. You must log-in using the actual SYS password. Do not connect using:
>sqlplus / as sysdba This will not work and CSALTER will fail, because this actually produces
two SQL*Plus sessions. Only one session must be running when CSALTER is executed, so you
must connect as follows:
>sqlplus sys/******** as sysdba
4. ALTER SYSTEM ENABLE RESTRICTED SESSION
5. SQL> @csalter.plb
CSALTER will prompt you to verify that you want to proceed to migrate the character set:

This script will update the content of the Oracle Data Dictionary.
Please ensure you have a good database backup before initiating this procedure.
Would you like to proceed (Y/N)?Y
It will begin by migrating the data dictionary objects and some other output will be displayed. Finally,
you should see the following:
PL/SQL procedure successfully completed.
Alter the database character set...
CSALTER operation completed, please restart database
At this point, you may shutdown and restart the database. When the database is back online, connect as
SYSDBA and verify the character set:
DBATEST> SELECT value FROM NLS_DATABASE_PARAMETERS WHERE parameter='NLS_CHARACTERSET';
VALUE
----------------------------------------------------------------------------------------------------------------------------------
AL32UTF8
So character set conversion is successfully completed. Now you must perform the post-conversion
steps.
Run Post-Conversion Checks
First check the Oracle alert log for any unusual errors or messages related to CSALTER. What you should
see is something like the following:
ALTER SYSTEM SET aq_tm_processes=0 SCOPE=MEMORY;
OS Pid: 54657126 executed alter system set events '22838 TRACE NAME CONTEXT LEVEL 1,
FOREVER'
alter database character set internal_use AL32UTF8
Updating character set in controlfile to AL32UTF8
Synchronizing connection with database character set information
Thu May 21 14:18:30 2015
Refreshing type attributes with new character set information
Completed: alter database character set internal_use AL32UTF8
If you see any other messages that indicate a problem, you will need to refer to the Oracle
Documentation referenced in Appendix A.
Run Post-Conversion Query
Now run the post_cs_mig.sql script, which will run a final check on all column data types. You should
see the following output:
DBATEST> @post_cs_mig.sql

CHARACTERSET TYPES_USED_IN
---------------------------------------- --------------------------------
AL16UTF16 NCHAR
AL16UTF16 NCLOB
AL16UTF16 NVARCHAR2
AL32UTF8 CHAR
AL32UTF8 CLOB
AL32UTF8 LONG
AL32UTF8 VARCHAR2
7 rows selected.
This query should return exactly 7 rows. If this select returns more than 7 rows (for example, 2 different
character sets for VARCHAR2) then there is something wrong and you will have errors in the alert log
during the conversion. Again, in this case, you will need to refer to the Oracle documentation.
NOTE: Even though the output shows AL16UTF16 for N-datatypes, this is correct and expected for
AL32UTF8 conversion.
Rebuild Catalog and Compile Objects
Character set conversion is basically an upgrade to the database. You must, therefore, rebuild the
catalog, rebuild Oracle packages and procedures, and recompile. Do the following:
2. STARTUP UPGRADE
3. From $ORACLE_HOME/rdbms/admin, run the following (be sure to spool the output):
catalog.sql
catproc.sql
utlrp.sql
4. Check the output logs for any ORA- errors. There should be none.
6. STARTUP
Now verify that everything in the DBA_REGISTRY is showing as VALID:
DBATEST> SELECT r.comp_name, r.version, r.status FROM dba_registry r;
COMP_NAME
--------------------------------------------------------------------------------------
--------------------------------------------
VERSION STATUS
------------------------------ --------------------------------------------
Oracle XML Database
11.2.0.3.0 VALID
Oracle Database Catalog Views
11.2.0.3.0 VALID

Oracle Database Packages and Types
11.2.0.3.0 VALID
Finally, run the chk_invalid_objects.sql script and you should not see any SYS or XDB invalid objects:
DBATEST> @chk_invalid_objects.sql
-------------------- ------------------------------ ------------------------------ -------
BATCHODM RESET_SEQUENCES PROCEDURE INVALID
C2C_APP CTRL_PROP_PKG PACKAGE BODY INVALID
C2C_APP PRICING_PKG_20141125 PACKAGE BODY INVALID
C2C_APP INVOICE_PKG_20140402 PACKAGE BODY INVALID
CIMS_APP CRTRDG_ORD_STAT_UPD_TRG TRIGGER INVALID
CMCPORTAL RTLCATAVAILABILITY_P PACKAGE BODY INVALID
MASTER MCLU_SBE_TEST VIEW INVALID
ORDERS_APP SYS_PLSQL_78791_68_1 TYPE INVALID
SSR SYS_PLSQL_129399_271_1 TYPE INVALID
Some of the above objects were invalid before the character set migration and some will become valid
when you complete the post-migration clean-up steps. You may want to point these out to the
application team(s). Otherwise, the database is now prepared for you to run the Data Pump import.
Perform Data Pump Import
The scripts that are included with this document contain an import parfile for the dump file from Step 7.
Obviously, you may need to modify this based on the database, but here is a fragment from our test
case:
USERID='/ as sysdba'
TABLES=(AS_SCHEDULER.ASAPP_CBD_PARMS,
AS_SCHEDULER.GJF_CHERYL1,
AS_SCHEDULER.NTR_IMPORT_MFG_EMAILS,
AS_SCHEDULER.PMDB_PROJECTS,
AS_SCHEDULER.PMDB_PROJECTS_0917,
AS_SCHEDULER.PRD_PROGRAMMCLUS,
.
.
.
CONTENT=DATA_ONLY
TABLE_EXISTS_ACTION=TRUNCATE
DIRECTORY=CSSCAN_DIR
DUMPFILE=expdp_csscan_tbls.dmp
JOB_NAME=impdp_csscan
LOGFILE=impdp_csscan_tbls.log
Since we truncated these tables in an earlier step, the TABLE_EXISTS_ACTION parameter may not be
necessary, but it’s included here as best practice. There is a corresponding shell script with which you
can perform the import.

> nohup impdp_csscan.sh &
Errors During Import
Although CSSCAN is a good tool, it is not 100% foolproof. Particularly, for data where column expansion
was required. So you may see the following type of errors in the Data Pump import log:
KUP-11007: conversion error loading table "MASTER"."LOCALE_FORMAT_T"
ORA-12899: value too large for column CURRENCY_SYMBOL_TXT (actual: 7, maximum: 5)
KUP-11009: data for row: CURRENCY_SYMBOL_TXT : 0X'2080202020'
As the error implies, CSSCAN missed this table column for column expansion. The solution is:
1. Truncate the table
2. Alter the table and expand the column size at least to the actual size specified
3. Re-run the Data Pump import for this table
There should not be many of these. For our test database (which is a clone of TOCUSQA3), there were
only two tables that had this issue.
Once the Data Pump import has completed successfully, you must then run the additional import using
the old imp command to reload the XML data, as we covered in Step 7: Export Convertible Table Data.
Re-Enable Constraints, Triggers, and Final Clean-Up
This is the final phase of character set conversion. You must now re-enable foreign and primary key
constraints and triggers and perform some final clean-up steps.
Enable Primary Key Constraints
You must enable primary keys first, before you can enable foreign keys, due to validation. To enable the
primary keys that were disabled in Step 8, run the gen_enable_global_pk.sql script. This will enable
all primary (and unique) keys for all tables based on the schema list created in an earlier step. This may
take a while to run, so use the gen_enable_global_pk.sh shell script to run this in the background.
Once this has completed, check the log for errors (there should be none).
Enable Foreign Key Constraints
Once the primary (and unique) key constraints are enabled, run the chk_fk_constraints.sql script to
look at the list of all FK constraints (based on schema_list) and their status. You will see that all FK
constraints are DISABLED and NOT VALIDATED.
Now run gen_enable_global_fk.sql to enable all FK constraints. This could take a while since there
will be many foreign keys and also because Oracle will be validating the FK data. So you may want to run
this in the background using the gen_enable_global_fk.sh shell script.

Once this has completed, execute: grep –i ora- enable_global_fk.log and you may see that some
FK’s could not be validated due to non-existent parent keys:
ORA-02298: cannot validate (C2C_APP.BUS_SEG_T_ACCT_T_FK) - parent keys not found
ORA-02298: cannot validate (C2C_APP.CONTACT_T_CON_CONTACT_T_FK) - parent keys not found
ORA-02298: cannot validate (C2C_APP.BUS_SEG_T_DOC_TYP_T_FK) - parent keys not found
ORA-02298: cannot validate (C2C_APP.BILL_BRND_DESCR_PROD_CD_FK) - parent keys not found
ORA-02298: cannot validate (C2C_APP.BILL_T_SLS_HIER_T_FK) - parent keys not found
ORA-02298: cannot validate (MASTER.SCHEDULE_REQ_ID_NBR_FK) - parent keys not found
ORA-02298: cannot validate (MASTER.STR_INI_IMPORT_STR_NBR_FK) - parent keys not found
In this case some of these FKs should be enabled with the NOVALIDATE option. Run the
gen_enable_global_fk_novalid.sql script to enable FKs without further validation. Then, run the
check script again. You will now see that all FKs are ENABLED, but not all are VALIDATED.
Again, depending on the database, you may want to engage the application team(s) to clean up the
orphaned keys (particularly, if this is Production).
Enable Triggers
Now enable the triggers for the tables that were exported/imported for convertible character set data.
Run the gen_enable_trig.sql script to enable the triggers. As always, check the output log for errors
(there should be none).
Reset Run Stats History Retention and Run Database Statistics
After character set conversion, set the statistics history retention back to the original value:
EXEC DBMS_STATS.ALTER_STATS_HISTORY_RETENTION(31);
The final step for character set conversion is to the gather database statistics using the DBMS package:
EXEC DBMS_STATS.GATHER_DATABASE_STATS(DEGREE=> 'n');
Where n is the CPU count. Since this can be time-consuming, use the gather_db_stats.sh shell script
to run this in background using nohup.
Reset Column Comments
Using the script(s) generated from an earlier step, add the comments back to the respective tables
where comments were cleared. For example:
DBATEST> @add_comments-INCENTIVE_BLIND_T.sql
Comment created.

Recreate Special Objects
In an earlier step, you saved off any packages, package bodies, procedures, etc. that would not covert
properly. You must now recreate these objects. For example, we saved off the
CIMS_APP.BATCH_FILE_PKG package, so now run the batch_file_pkg.sql to recreate it. For example:
DBATEST> @batch_file_pkg.sql
Package body created.
DBATEST>
DBATEST> GRANT EXECUTE ON CIMS_APP.BATCH_FILE_PKG TO CIMS_BATCH;
Grant succeeded.
DBATEST>
DBATEST> spool off
DBATEST> quit
TIP: Be sure to SET DEFINE OFF when recreating these objects.
Drop Intermediate Tables
The final step is to drop the intermediate tables created in SYS that were used to run the various scripts
for character set conversion.
DBATEST> drop table schema_list purge;
Table dropped.
DBATEST> drop table export_list purge;
Table dropped.
Uninstall CSSCAN
Uninstalling CSSCAN is a simple two-step process (log-in as SYSDBA):
drop user CSMIG cascade;
drop tablespace CSMIG including contents and datafiles;
Conclusion
Character Set conversion can be a time-consuming and cumbersome process. In addition, the level of
complexity is dependent on the size and type of the data needing conversion. The Oracle tools (and
attached scripts) should make this process somewhat easier. But there will be cases in which you will
need to rely on the documents in Appendix A. The Oracle documentation is very good, but rather dense.
That is why we’ve tried here to capture a step-by-step approach.

Appendix A
Oracle Support Doc ID 745809.1: Installing and configuring CSSCAN in 10g and 11g
Oracle Support Doc ID 260192.1: Changing the NLS_CHARACTERSET to AL32UTF8 / UTF8 (Unicode) in 8i,
9i , 10g and 11g
Oracle Support Doc ID 258904.1: Solving Convertible or Lossy data in Data Dictionary objects reported
by CSSCAN when changing the NLS_CHARACTERSET

Converting the Oracle Database Character Set

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Converting the Oracle Database Character Set

Similar to Converting the Oracle Database Character Set (20)

Converting the Oracle Database Character Set