6. Data Loading
IO Requirement
• 1 Terabyte per hour
• 17 GByte/minute
• 291 MByte/s
• With compressed source data
• 1057 Gbyte Text Files 136 Gbyte GZIP Files
• 2.3 GByte/minute
• 40 MByte/s
7. Data Loading
CPU/Memory Requirement
• Data loading is CPU/Memory Constrained
• Raw is the cheapest
• Timestamp is the most expensive
• One SQL*Loader will use one copy of in-memory meta data
8. Data Loading
Compress the Data
• Use preprocessor in external table definition
9. Data Loading
External Table
External Table
create table et_target Definition
(
column definition list ... Reference the
) Mount Point
organization external
(type oracle_loader Uncompress
default directory SPEEDY_FILESYSTEM the data
preprocessor exec_file_dir: 'zcat' options '-c’
characterset 'ZHS16GBK’
The Characterset
badfile ERROR_DUMP: ’FAST_LOAD.bad’
must match the
logfile ERROR_DUMP: ’FAST_LOAD.log’
Characterset of the
(
file column mapping list ...
Files
) Note Compressed
location Files
(file_1.gz, file_2.gz, file_3.gz, file_4.gz )
reject limit 1000 Parallel should
parallel 4 match the
/ number of Files
10. Data Loading
How to Use
alter session enable parallel dml;
alter table target parallel;
alter table et_target parallel;
insert /*+ append */ into target
select * from et_target;
11. Data Loading
Live Demo
• Load 1 TB data
TABLE SIZE OF SOURCE DATA NUMBER OF RECORDS
• Transactions • 52 GByte • 461M
• Payments • 54 GByte • 461M
• Line Items • 936 Gbyte • 6945M
• Total • 1042 GByte • 7867M
13. Data Unloading
What is it look like?
• Methods Available?
• exp?
• Data Pump?
• sqlplus spool?
• RMAN?
• sqluldr2 by anysql.net?
• Others?
• Expected Speed?
• CPU-bound or IO-bound?
16. Data Unloading
IO Requirement
• 1 Terabyte per hour
• 17 GByte/minute
• 291 MByte/s
• With compressed target data
• 1057 Gbyte Text Files 136 Gbyte GZIP Files
• 2.3 GByte/minute
• 40 MByte/s
17. Data Unloading
CPU Requirement
• Data unloading is CPU Constrained
• Raw is the cheapest
• Timestamp is the most expensive
18. Data Unloading
Compress the Data
• Use FIFO Pipe file to compress the data
19. Data Unloading
Live Demo
• Unload 1 TB data
TABLE SIZE OF SOURCE DATA NUMBER OF RECORDS
• Transactions • 52 GByte • 461M
• Payments • 54 GByte • 461M
• Line Items • 936 Gbyte • 6945M
• Total • 1042 GByte • 7867M
