1. Oracle Exadata
Bottlenecks, Bottlenecks, and more Bottlenecks:
Lessons Learned from 2 Years of Exadata Benchmarks
Tyler D Muth
Chief Database Engineer
Oracle Public Sector
tylermuth.wordpress.com

3. I only used the most challenging POVs as examples. Some,
I talked a lot about the challenges. In each my examples the
ultimate result was very successful.

4. Tangent 1
(is it a tangent if you schedule it?)
Mixed Workload (show (next slide))
½ rack
~15,000 transactions per second from swingbench
Mix of read & write OLTP
what this machine can do in isolation) of physical
I/O every few seconds
DW queries have little or no impact on TPS metric

6. Reality Check
Physics govern the performance of
computing systems
Marketing cannot change the laws of
physics
Very few people in the world understand
the components end-to-end

7. Hard Drive Capacity vs Throughput
120,000
120000
100000
90,000
80,000
Factor of Change
80000
60000
Capacity Factor of Change
Throughput Factor of Change
40000
20000
1 1 33 41 72
0
ST506 (1979) Cheetah 15k.6 (2008) Cheetah 15k.7 (2011) Pulsar XT.2 SSD (2011)
Seagate Hard Drive Model (Year)
ST506 Cheetah 15k.6 Cheetah 15k.7 Pulsar XT.2 SSD
(1979) (2008) (2011) (2011)
Capacity (GB) 0.005 450 600 400
Throughput (MB/s) 5 164 204 360

8. List of Device Bit Rates
wikipedia - http://sn.im/24mqy8q
Device Bandwidth Move 1 TB
Ethernet 10 | 100 Mbit 1.25 | 12.5 MB/s 9+ days | 18 hours
Ethernet 1 | 10 Gbit 125 MB/s | 1.25 GB/s 2.3 hours | 13.7 min
Infiniband QDR 4× 4 GB/s 4.3 min
PCI Express 2.0 4 GB/s 4.3 min
USB Hi-Speed (USB 2.0) 60 MB/s 7.2 hours
Fibre Channel 2 | 4 | 8 GFC 212.5 | 425 | 850 MB/s 82 min | 41 min | 20.5 min
7200 RPM Drive 155 MB/s 112 min
15k RPM Drive 204 MB/s 85 min
Enterprise SSD 360 MB/s 48 min

9. Throughput GB/s
80.0
75
70.0
60.0
2 Gbps Fibre Channel x2
50.0
4 Gbps Fibre Channel x2
40.0
8 Gbps Fibre Channel x2
30.0 37
Exadata 1/4 - Disk
25
20.0 Exadata 1/2 - Disk
10.0 12.5 16 Exadata Full - Disk
0.4 0.8 5.4 Exadata 1/4 - Disk & Flash
1.6
0.0
Exadata 1/2 - Disk & Flash
Exadata Full - Disk & Flash

10. Basic Capacity Planning
Basic Capacity Planning
A transaction takes 10 CPU Milli Seconds of CPU
and performs 5 random Disk I/Os
The requirement is to support 1,000
Transactions/Sec
This means
#CPU cores needed =
CPU secs/tx x Transaction Rate / Max Utilization
0.01 x 1000 / 0.65 = 15.38 round up to 16
#Disk Spindles Needed =
IOPS/tx x Transaction rate / IOP per disk
5 X 1000 / 35 = 142 Disks

11. Insert Performance 500k Rows
Indexes Indexes
Enabled Disabled
Individual Inserts 52.1 seconds 37.9 seconds
Bulk Inserts 4.2 seconds 2.4 seconds

14. Monitoring Tools
Macro
AWR
Single-node / Global / Compare Period / Single SQL Statement (Color SQL)
Mining with R Look at my blog shortly, also karlarao.wordpress.com
AWR Formatter (show)
EM Avg Act Sessions
OSWatcher
CPU Monitor Dominic Giles
DB Time Viewer
EM 12c
Micro
ASH
SQLT MOS Note: 215187.1 (show):
http://sn.im/24ocbsn - Main
http://sn.im/24occct - Compare
Snapper Tanel Poder - http://sn.im/tanel-snapper

15. PL/SQL Hierarchical Profiler
NAME FUNCTION SUBTREE_ELAPSED_TIME FUNCTION_ELAPSED_TIME CALLS
---------------------------------- ------------------------- -------------------- --------------------- ----------
MDSYS.SDO_GEOM SDO_UNION 339697300 19 1
MDSYS.SDO_GEOM GET_DIM_ARRAY 187 187 2
MDSYS.SDO_GEOM SDO_UNION 339697094 526 1
MDSYS.SDO_3GL GEOM_OPERATION 339696568 71518056 1
. __plsql_vm@1 435 10 3
MDSYS.SDO_CS DISAMBIGUATE_PROJ_NAME 640 241 2
MDSYS.SDO_CS __static_sql_exec_line472 399 399 2
MDSYS.SDO_3GL VALIDATE_GEOM 268178077 268177852 1
. __plsql_vm@1 225 10 1
MDSYS.SDO_CS DISAMBIGUATE_PROJ_NAME 640 241 2
MDSYS.SDO_CS __static_sql_exec_line472 399 399 2
MDSYS.SDO_GEOM __pkg_init 10 10 1
MDSYS.SDO_UTIL APPEND 208234 208234 2
MDSYS.SDO_UTIL INTERNAL_MERGE_LINESTRING 317104843 78041429 1
MDSYS.SDO_UTIL CONCAT_LINES 238540586 86537313 12314
MDSYS.SDO_UTIL INTERNAL_ORDINATE_COPY 151276858 151276858 80904
MDSYS.SDO_UTIL INTERNAL_REVERSE_LINE_POI 611137 611137 230
MDSYS.SDO_UTIL REVERSE_LINESTRING 115278 67102 3760
MDSYS.SDO_UTIL INTERNAL_REVERSE_LINE_POI 48176 48176 3760
MDSYS.SDO_UTIL TRUNCATE_NUMBER 522828 522828 198680
MDSYS.SDO_UTIL __pkg_init 11 11 1
SYS.DBMS_ASSERT QUALIFIED_SQL_NAME 195 195 1
SYS.DBMS_ASSERT SIMPLE_SQL_NAME 95 95 1
MDSYS.SDO_TOPO_GEOMETRY __dyn_sql_exec_line1048 6718375 6718375 1
MDSYS.SDO_TOPO_GEOMETRY __dyn_sql_exec_line978 3986 3986 1
MDSYS.SDO_TOPO_GEOMETRY __sql_fetch_line1054 2482917 2482917 12316

16. Google can be your enemy
Large SGA Example
Show exacheck: http://sn.im/24ocb99

17. Hints Are Evil
Hints that define specific access paths apply to your CURRENT platform
create or replace trigger disable_hints
after logon on database
declare
x varchar2(200) := q'! alter session set "_optimizer_ignore_hints" = TRUE !';
l_program varchar2(1000);
begin
if sys_context('USERENV','SESSION_USER') = 'FOO' then
select upper(program) into l_program
from v$session
where sys_context('USERENV','SESSIONID') = AUDSID;
if l_program like 'JDBC THIN%' then
execute immediate x;
end if;

18. Average Active Sessions by Wait Class
(scale of 0-60 sessions of a possible 256)
60
Average Active Sessions (256 would be 100% busy)
50
40
30 USER_IO
OTHER
DB_CPU
CONCURRENCY
20
CLUSTER
APPLICATION
10
0
Snapshot Date / Time (PDT)

19. OS Average CPU % by Database Node
(0-100 % Scale)
100
90
80
70
60
50
40
30
20
10
0 Node 1
Node 2
Node 1

20. Case Object Tracking
Track (insert) 21,000 new objects per
second
2,200 tracking queries per second
2 years = 72 TB of data
X2-
Data coming in from flat files, new file
every few seconds

21. No need for indexes?
#CPU cores needed =
CPU secs/tx x Transaction Rate / Max Utilization
Guess the fastest avg smartscan = 0.5 s
0.5 * 2200 / 0.65 = 1,692 cores !?!?
0.04 * 2200 / 0.65 = 135 cores

22. No time-based partitioning scheme
Query response requirements = global
indexes
1 index on 1 column took 4+ days and 24
TB of temp
(show SQL Monitor)

23. Insert Rate Indexes = bad
Bulk inserts, no indexes = 1.1 million rows
per second
With 1 index on 1 column = 12,000 rows
per second

24. Case 1 DDL = Expensive
Create new external table for each new
file
Create & delete about 30 per minute
Most of the time spend in DDL
Solution:

25. DDL Solution
01 loop
02 exit when l_current_file is null;
03 l_location_string := l_location_string ||
''''||p_file_array(l_current_file)||''',';
04 l_current_file := p_file_array.next(l_current_file);
05 end loop;
06
07 l_location_string := rtrim(l_location_string,',');
08 log('Location ['||l_random_string||'] maps to
['||substr(l_location_string,1,900)||']');
09 q := 'ALTER TABLE '||g_ext_tab_name||' LOCATION ('||l_location_string||')' ;
10 log(substr(q,1,900));
11 execute immediate q;

26. Index Write IOPs
Inserting ~12,000 rows per second
Required 22,000+
OSWatcher iostat files indicated 27,000+
write IOPs uh oh
Metrics in dba_hist_sysmetric_summary
Achieved 44,000+
Show write-IOPs PDF

27. 1G
1G
1.4 1G
1.2
Service Time (ms)
1.0 0G
0.8
0.6
0.4
type
a
G Disk
272,154 G
a
G Flash
258,475 G
250,000
200,000
150,000
Write IOPs
100,000
50,000
13,104 G
10,686 G
0
20:10 20:20 20:30 20:40 20:50 21:00
date_stamp

28. Dynamic Remastering
Show AWR Mining PDF

29. Average Active Sessions (AAS) for U
2000 1943.41
G
Other
1500 Cluster
Network
Administrative
Configuration
Commit
Application
1000 Concurrency
System I/O
User I/O
Scheduler
CPU
500
CPU Cores
392.19
G
185
140.84 153.77 G
G
G
77.42 97.72 88.65
G G
G
0.26
0 G
Fri Sa Su Mo Tu We Th Fri Sa
,F t, F n, n, e, d u, ,M t, M
eb eb Fe Fe Fe ,F Ma ar ar
24 25 b b b 28 eb r0 02 03
26 27 29 1

30. Average Active Sessions by Wait Class − U
Only Values >= 95th Percentile
1200
1000
Other
800
600
400
200 132.6 131.12
25.99 G G 17.74
0 G G
1200
1000 795.65
Cluster
800 G
600
400
200 34.89 72.79
G 37.91
G G
0
1200
Configuration
1000
800
600
400 214.79
200 G
20.36
0 G
1200
1000
Commit
800
600
400
200 18.65 22.5 25.8
0 G G G
1200 1096.84
Concurrency
G
1000
800
600
400
200 20.49 92.7
G
0 G
1200
1000
User I/O
800
600
400 250.64
200 79.29 60.86 77.33 51.57
G 134.34 76.81 62.79
G G G G
G G G
0
1200
1000
800
CPU
600
400
200 55.9 17.73 43.37 41.23
G G G
0 G
780
785
790
795
800
805
810
815
820
825
830
835
840
845
850
855
860
865
870
875
880
885
890
895
900
905
910
915
920
925
930
935
940
945
950
955
960
965
970
975
980
985
Sa Su Mo Tu We Th Fri Sa
t, F n, n, e, d u, ,M t, M
eb Fe Fe Fe ,F Ma ar ar
25 b b b 28 eb r0 02 03
26 27 29 1