Exadata下的数据并行加载、并行卸载及性能监控

About Me• 工作于Oracle Real World Database Performance Group• 关键词: 高性能, 高可扩展性, Exadata• 微博: http://www.weibo.com/kaiyaohuang• 博客: http://www.os2ora.com• 电子邮件：kaiyao.huang@gmail.com• 电话： 186-649-33229/黄凯耀

Agenda• Data Loading• Data Unloading• Performance Monitoring

Data Loading What is it look like?• Methods Available? • imp? • Data Pump? • SQL * Loader? • RMAN? • External Table? • Others?• Expected Speed?• CPU-bound or IO-bound?

Architecture

Data Loading IO Requirement• 1 Terabyte per hour • 17 GByte/minute • 291 MByte/s• With compressed source data • 1057 Gbyte Text Files  136 Gbyte GZIP Files • 2.3 GByte/minute • 40 MByte/s

Data Loading CPU/Memory Requirement• Data loading is CPU/Memory Constrained • Raw is the cheapest • Timestamp is the most expensive • One SQL*Loader will use one copy of in-memory meta data

Data Loading Compress the Data• Use preprocessor in external table definition

Data Loading External Table External Tablecreate table et_target Definition(column definition list ... Reference the) Mount Pointorganization external(type oracle_loader Uncompressdefault directory SPEEDY_FILESYSTEM the datapreprocessor exec_file_dir: zcat options -c’characterset ZHS16GBK’ The Charactersetbadfile ERROR_DUMP: ’FAST_LOAD.bad’ must match thelogfile ERROR_DUMP: ’FAST_LOAD.log’ Characterset of the(file column mapping list ... Files) Note Compressedlocation Files(file_1.gz, file_2.gz, file_3.gz, file_4.gz )reject limit 1000 Parallel shouldparallel 4 match the/ number of Files

Data Loading How to Usealter session enable parallel dml;alter table target parallel;alter table et_target parallel;insert /*+ append */ into targetselect * from et_target;

Data Loading Live Demo• Load 1 TB data TABLE SIZE OF SOURCE DATA NUMBER OF RECORDS • Transactions • 52 GByte • 461M • Payments • 54 GByte • 461M • Line Items • 936 Gbyte • 6945M • Total • 1042 GByte • 7867M

Data Unloading What is it look like?• Methods Available? • exp? • Data Pump? • sqlplus spool? • RMAN? • sqluldr2 by anysql.net? • Others?• Expected Speed?• CPU-bound or IO-bound?

Architecture

Data Unloading Parallel Table Function

Data Unloading IO Requirement• 1 Terabyte per hour • 17 GByte/minute • 291 MByte/s• With compressed target data • 1057 Gbyte Text Files  136 Gbyte GZIP Files • 2.3 GByte/minute • 40 MByte/s

Data Unloading CPU Requirement• Data unloading is CPU Constrained • Raw is the cheapest • Timestamp is the most expensive

Data Unloading Compress the Data• Use FIFO Pipe file to compress the data

Data Unloading Live Demo• Unload 1 TB data TABLE SIZE OF SOURCE DATA NUMBER OF RECORDS • Transactions • 52 GByte • 461M • Payments • 54 GByte • 461M • Line Items • 936 Gbyte • 6945M • Total • 1042 GByte • 7867M

Performance Monitoring OS Real-Time Monitoring• vmstat?• iostat?• top?• sar?• netstat?• collectl?

Architecture of collectl

mon cluster-wide collectl

Performance Monitoring DB Monitoring• v$session?• v$sysstat?• statspack?• AWR?• ADDM?• SQL Monitor Report?

Performance Monitoring SQL Monitor Report• 第一次执行的SQL Monitor Report• 第二次执行的SQL Monitor Report

Live ExampleINSERT /*+ APPEND */INTO T_DSELECT *FROM T_A a,T_B b,T_C cWHERE a.id = b.idAND b.number = c.number;

Live Example

Reference• 史上最快，最具可扩展性的文本导出方法: http://www.os2ora.com/the-fastest- data-unload-method/• 史上最快，最具可扩展性的文本导入方法: http://www.os2ora.com/the-fastest- data-load-method-best-practice/• Linux操作系统资源监控工具重点推荐 — collectl: http://www.os2ora.com/linux- performance-monitoring-tool-recommend-collectl/• Oracle监控工具概览: http://www.os2ora.com/oracle-monitoring-tool-summary- and-recommend/• 测试: 一个SQL Monitor Report的具体例子: http://www.os2ora.com/test-sql- monitor-report-example/• 利用SQL Monitor Report对SQL进行诊断与调优: http://www.os2ora.com/use- sql-monitor-report-to-tune-and-diagnose-sql/

PX Architecture

PX topics…• Degree of Parallelism• PX Distribution• PX Statement Queuing• Hash Partition• Partition-Wise Join

http://www.os2ora.com	254
http://jiyzhang-cn	21
http://jiyzhang-cn.cn.oracle.com	15
http://jiyzhang-cn.us.oracle.com	12
http://oraclesmith.cn.oracle.com	5
http://xianguo.com	4
http://www.zhuaxia.com	2
http://10.182.38.222	2
http://zhuaxia.com	1
http://www.os2ora.com HTTP	1
http://localhost	1

Exadata下的数据并行加载、并行卸载及性能监控

by Kaya Huang on Aug 06, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

11 Embeds 318

Statistics

Exadata下的数据并行加载、并行卸载及性能监控 — Presentation Transcript