Oracle ExadataBottlenecks, Bottlenecks, and more Bottlenecks:Lessons Learned from 2 Years of Exadata BenchmarksTyler D Mut...
I only used the most challenging POVs as examples. Some,I talked a lot about the challenges. In each my examples theultima...
Tangent 1           (is it a tangent if you schedule it?)Mixed Workload     (show (next slide))  ½ rack  ~15,000 transacti...
Reality CheckPhysics govern the performance ofcomputing systemsMarketing cannot change the laws ofphysicsVery few people i...
Hard Drive Capacity vs Throughput                                                                                         ...
List of Device Bit Rates                                  wikipedia - http://sn.im/24mqy8qDevice                          ...
Throughput GB/s80.0                                                        7570.060.0                                     ...
Basic Capacity PlanningBasic Capacity Planning  A transaction takes 10 CPU Milli Seconds of CPU  and performs 5 random Dis...
Insert Performance            500k Rows                  Indexes         Indexes                  Enabled         Disabled...
Monitoring ToolsMacro   AWR        Single-node / Global / Compare Period / Single SQL Statement (Color SQL)        Mining ...
PL/SQL Hierarchical ProfilerNAME                                 FUNCTION                  SUBTREE_ELAPSED_TIME FUNCTION_E...
Google can be your enemyLarge SGA ExampleShow exacheck: http://sn.im/24ocb99
Hints Are Evil      Hints that define specific access paths apply to your CURRENT platformcreate or replace trigger disabl...
Average Active Sessions by Wait Class                                                                (scale of 0-60 sessio...
OS Average CPU % by Database Node                       (0-100 % Scale)100 90 80 70 60 50 40  30  20  10      0           ...
Case     Object TrackingTrack (insert) 21,000 new objects persecond2,200 tracking queries per second2 years = 72 TB of dat...
No need for indexes?#CPU cores needed =    CPU secs/tx x Transaction Rate / Max UtilizationGuess the fastest avg smartscan...
No time-based partitioning schemeQuery response requirements = globalindexes1 index on 1 column took 4+ days and 24TB of t...
Insert Rate     Indexes = badBulk inserts, no indexes = 1.1 million rowsper secondWith 1 index on 1 column = 12,000 rowspe...
Case 1    DDL = ExpensiveCreate new external table for each newfileCreate & delete about 30 per minuteMost of the time spe...
DDL Solution01 loop02       exit when l_current_file is null;03       l_location_string := l_location_string ||           ...
Index Write IOPsInserting ~12,000 rows per secondRequired 22,000+OSWatcher iostat files indicated 27,000+write IOPs uh ohM...
1G            1G      1.4                                                                                             1G  ...
Dynamic RemasteringShow AWR Mining PDF
Average Active Sessions (AAS) for U2000                                                                                   ...
Average Active Sessions by Wait Class − U                                                                                 ...
Bottlenecks, Bottlenecks, and more Bottlenecks: Lessons Learned from 2 Years of Exadata Benchmarks
Bottlenecks, Bottlenecks, and more Bottlenecks: Lessons Learned from 2 Years of Exadata Benchmarks
Bottlenecks, Bottlenecks, and more Bottlenecks: Lessons Learned from 2 Years of Exadata Benchmarks
Bottlenecks, Bottlenecks, and more Bottlenecks: Lessons Learned from 2 Years of Exadata Benchmarks
Bottlenecks, Bottlenecks, and more Bottlenecks: Lessons Learned from 2 Years of Exadata Benchmarks
Bottlenecks, Bottlenecks, and more Bottlenecks: Lessons Learned from 2 Years of Exadata Benchmarks
Bottlenecks, Bottlenecks, and more Bottlenecks: Lessons Learned from 2 Years of Exadata Benchmarks
Bottlenecks, Bottlenecks, and more Bottlenecks: Lessons Learned from 2 Years of Exadata Benchmarks
Bottlenecks, Bottlenecks, and more Bottlenecks: Lessons Learned from 2 Years of Exadata Benchmarks
Upcoming SlideShare
Loading in …5
×

Bottlenecks, Bottlenecks, and more Bottlenecks: Lessons Learned from 2 Years of Exadata Benchmarks

1,116 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,116
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Bottlenecks, Bottlenecks, and more Bottlenecks: Lessons Learned from 2 Years of Exadata Benchmarks

  1. 1. Oracle ExadataBottlenecks, Bottlenecks, and more Bottlenecks:Lessons Learned from 2 Years of Exadata BenchmarksTyler D MuthChief Database EngineerOracle Public Sectortylermuth.wordpress.com
  2. 2. I only used the most challenging POVs as examples. Some,I talked a lot about the challenges. In each my examples theultimate result was very successful.
  3. 3. Tangent 1 (is it a tangent if you schedule it?)Mixed Workload (show (next slide)) ½ rack ~15,000 transactions per second from swingbench Mix of read & write OLTP what this machine can do in isolation) of physical I/O every few seconds DW queries have little or no impact on TPS metric
  4. 4. Reality CheckPhysics govern the performance ofcomputing systemsMarketing cannot change the laws ofphysicsVery few people in the world understandthe components end-to-end
  5. 5. Hard Drive Capacity vs Throughput 120,000 120000 100000 90,000 80,000Factor of Change 80000 60000 Capacity Factor of Change Throughput Factor of Change 40000 20000 1 1 33 41 72 0 ST506 (1979) Cheetah 15k.6 (2008) Cheetah 15k.7 (2011) Pulsar XT.2 SSD (2011) Seagate Hard Drive Model (Year) ST506 Cheetah 15k.6 Cheetah 15k.7 Pulsar XT.2 SSD (1979) (2008) (2011) (2011) Capacity (GB) 0.005 450 600 400 Throughput (MB/s) 5 164 204 360
  6. 6. List of Device Bit Rates wikipedia - http://sn.im/24mqy8qDevice Bandwidth Move 1 TBEthernet 10 | 100 Mbit 1.25 | 12.5 MB/s 9+ days | 18 hoursEthernet 1 | 10 Gbit 125 MB/s | 1.25 GB/s 2.3 hours | 13.7 minInfiniband QDR 4× 4 GB/s 4.3 minPCI Express 2.0 4 GB/s 4.3 minUSB Hi-Speed (USB 2.0) 60 MB/s 7.2 hoursFibre Channel 2 | 4 | 8 GFC 212.5 | 425 | 850 MB/s 82 min | 41 min | 20.5 min7200 RPM Drive 155 MB/s 112 min15k RPM Drive 204 MB/s 85 minEnterprise SSD 360 MB/s 48 min
  7. 7. Throughput GB/s80.0 7570.060.0 2 Gbps Fibre Channel x250.0 4 Gbps Fibre Channel x2 40.0 8 Gbps Fibre Channel x2 30.0 37 Exadata 1/4 - Disk 25 20.0 Exadata 1/2 - Disk 10.0 12.5 16 Exadata Full - Disk 0.4 0.8 5.4 Exadata 1/4 - Disk & Flash 1.6 0.0 Exadata 1/2 - Disk & Flash Exadata Full - Disk & Flash
  8. 8. Basic Capacity PlanningBasic Capacity Planning A transaction takes 10 CPU Milli Seconds of CPU and performs 5 random Disk I/Os The requirement is to support 1,000 Transactions/SecThis means #CPU cores needed = CPU secs/tx x Transaction Rate / Max Utilization 0.01 x 1000 / 0.65 = 15.38 round up to 16 #Disk Spindles Needed = IOPS/tx x Transaction rate / IOP per disk 5 X 1000 / 35 = 142 Disks
  9. 9. Insert Performance 500k Rows Indexes Indexes Enabled DisabledIndividual Inserts 52.1 seconds 37.9 secondsBulk Inserts 4.2 seconds 2.4 seconds
  10. 10. Monitoring ToolsMacro AWR Single-node / Global / Compare Period / Single SQL Statement (Color SQL) Mining with R Look at my blog shortly, also karlarao.wordpress.com AWR Formatter (show) EM Avg Act Sessions OSWatcher CPU Monitor Dominic Giles DB Time Viewer EM 12cMicro ASH SQLT MOS Note: 215187.1 (show): http://sn.im/24ocbsn - Main http://sn.im/24occct - Compare Snapper Tanel Poder - http://sn.im/tanel-snapper
  11. 11. PL/SQL Hierarchical ProfilerNAME FUNCTION SUBTREE_ELAPSED_TIME FUNCTION_ELAPSED_TIME CALLS---------------------------------- ------------------------- -------------------- --------------------- ---------- MDSYS.SDO_GEOM SDO_UNION 339697300 19 1 MDSYS.SDO_GEOM GET_DIM_ARRAY 187 187 2 MDSYS.SDO_GEOM SDO_UNION 339697094 526 1 MDSYS.SDO_3GL GEOM_OPERATION 339696568 71518056 1 . __plsql_vm@1 435 10 3 MDSYS.SDO_CS DISAMBIGUATE_PROJ_NAME 640 241 2 MDSYS.SDO_CS __static_sql_exec_line472 399 399 2 MDSYS.SDO_3GL VALIDATE_GEOM 268178077 268177852 1 . __plsql_vm@1 225 10 1 MDSYS.SDO_CS DISAMBIGUATE_PROJ_NAME 640 241 2 MDSYS.SDO_CS __static_sql_exec_line472 399 399 2 MDSYS.SDO_GEOM __pkg_init 10 10 1 MDSYS.SDO_UTIL APPEND 208234 208234 2 MDSYS.SDO_UTIL INTERNAL_MERGE_LINESTRING 317104843 78041429 1 MDSYS.SDO_UTIL CONCAT_LINES 238540586 86537313 12314 MDSYS.SDO_UTIL INTERNAL_ORDINATE_COPY 151276858 151276858 80904 MDSYS.SDO_UTIL INTERNAL_REVERSE_LINE_POI 611137 611137 230 MDSYS.SDO_UTIL REVERSE_LINESTRING 115278 67102 3760 MDSYS.SDO_UTIL INTERNAL_REVERSE_LINE_POI 48176 48176 3760 MDSYS.SDO_UTIL TRUNCATE_NUMBER 522828 522828 198680 MDSYS.SDO_UTIL __pkg_init 11 11 1 SYS.DBMS_ASSERT QUALIFIED_SQL_NAME 195 195 1 SYS.DBMS_ASSERT SIMPLE_SQL_NAME 95 95 1 MDSYS.SDO_TOPO_GEOMETRY __dyn_sql_exec_line1048 6718375 6718375 1 MDSYS.SDO_TOPO_GEOMETRY __dyn_sql_exec_line978 3986 3986 1 MDSYS.SDO_TOPO_GEOMETRY __sql_fetch_line1054 2482917 2482917 12316
  12. 12. Google can be your enemyLarge SGA ExampleShow exacheck: http://sn.im/24ocb99
  13. 13. Hints Are Evil Hints that define specific access paths apply to your CURRENT platformcreate or replace trigger disable_hintsafter logon on databasedeclare x varchar2(200) := q! alter session set "_optimizer_ignore_hints" = TRUE !; l_program varchar2(1000);begin if sys_context(USERENV,SESSION_USER) = FOO then select upper(program) into l_program from v$session where sys_context(USERENV,SESSIONID) = AUDSID; if l_program like JDBC THIN% then execute immediate x; end if;
  14. 14. Average Active Sessions by Wait Class (scale of 0-60 sessions of a possible 256) 60Average Active Sessions (256 would be 100% busy) 50 40 30 USER_IO OTHER DB_CPU CONCURRENCY 20 CLUSTER APPLICATION 10 0 Snapshot Date / Time (PDT)
  15. 15. OS Average CPU % by Database Node (0-100 % Scale)100 90 80 70 60 50 40 30 20 10 0 Node 1 Node 2 Node 1
  16. 16. Case Object TrackingTrack (insert) 21,000 new objects persecond2,200 tracking queries per second2 years = 72 TB of dataX2-Data coming in from flat files, new fileevery few seconds
  17. 17. No need for indexes?#CPU cores needed = CPU secs/tx x Transaction Rate / Max UtilizationGuess the fastest avg smartscan = 0.5 s0.5 * 2200 / 0.65 = 1,692 cores !?!?0.04 * 2200 / 0.65 = 135 cores
  18. 18. No time-based partitioning schemeQuery response requirements = globalindexes1 index on 1 column took 4+ days and 24TB of temp(show SQL Monitor)
  19. 19. Insert Rate Indexes = badBulk inserts, no indexes = 1.1 million rowsper secondWith 1 index on 1 column = 12,000 rowsper second
  20. 20. Case 1 DDL = ExpensiveCreate new external table for each newfileCreate & delete about 30 per minuteMost of the time spend in DDLSolution:
  21. 21. DDL Solution01 loop02 exit when l_current_file is null;03 l_location_string := l_location_string || ||p_file_array(l_current_file)||,;04 l_current_file := p_file_array.next(l_current_file);05 end loop;0607 l_location_string := rtrim(l_location_string,,);08 log(Location [||l_random_string||] maps to [||substr(l_location_string,1,900)||]);09 q := ALTER TABLE ||g_ext_tab_name|| LOCATION (||l_location_string||) ;10 log(substr(q,1,900));11 execute immediate q;
  22. 22. Index Write IOPsInserting ~12,000 rows per secondRequired 22,000+OSWatcher iostat files indicated 27,000+write IOPs uh ohMetrics in dba_hist_sysmetric_summaryAchieved 44,000+Show write-IOPs PDF
  23. 23. 1G 1G 1.4 1G 1.2 Service Time (ms) 1.0 0G 0.8 0.6 0.4 type a G Disk 272,154 G a G Flash 258,475 G250,000200,000150,000 Write IOPs100,00050,000 13,104 G 10,686 G 0 20:10 20:20 20:30 20:40 20:50 21:00 date_stamp
  24. 24. Dynamic RemasteringShow AWR Mining PDF
  25. 25. Average Active Sessions (AAS) for U2000 1943.41 G Other1500 Cluster Network Administrative Configuration Commit Application1000 Concurrency System I/O User I/O Scheduler CPU 500 CPU Cores 392.19 G 185 140.84 153.77 G G G 77.42 97.72 88.65 G G G 0.26 0 G Fri Sa Su Mo Tu We Th Fri Sa ,F t, F n, n, e, d u, ,M t, M eb eb Fe Fe Fe ,F Ma ar ar 24 25 b b b 28 eb r0 02 03 26 27 29 1
  26. 26. Average Active Sessions by Wait Class − U Only Values >= 95th Percentile12001000 Other 800 600 400 200 132.6 131.12 25.99 G G 17.74 0 G G12001000 795.65 Cluster 800 G 600 400 200 34.89 72.79 G 37.91 G G 01200 Configuration1000 800 600 400 214.79 200 G 20.36 0 G12001000 Commit 800 600 400 200 18.65 22.5 25.8 0 G G G1200 1096.84 Concurrency G1000 800 600 400 200 20.49 92.7 G 0 G12001000 User I/O 800 600 400 250.64 200 79.29 60.86 77.33 51.57 G 134.34 76.81 62.79 G G G G G G G 012001000 800 CPU 600 400 200 55.9 17.73 43.37 41.23 G G G 0 G 780 785 790 795 800 805 810 815 820 825 830 835 840 845 850 855 860 865 870 875 880 885 890 895 900 905 910 915 920 925 930 935 940 945 950 955 960 965 970 975 980 985 Sa Su Mo Tu We Th Fri Sa t, F n, n, e, d u, ,M t, M eb Fe Fe Fe ,F Ma ar ar 25 b b b 28 eb r0 02 03 26 27 29 1

×