Hotsos Symposium 2018-Proving out flash storage array performance using swingbench and slob

1. Proving out Flash Storage Array Performance using Swingbench and SLOB
Kapil Goyal, Fidelity Investments
@kapil_goyal

2. Who am I?
 Lead for Oracle database performance team at Fidelity Investments
 Worked with Oracle Consulting and got exposure to many different
companies' database performance challenges
 Speaker at ECO, BIWA, DOUG, Hostos, Enkitec Expo, IOUG and AIOUG
 Published articles for DOUG and IOUG SELECT magazine
 Certified OCP DBA 8i to 11g

3. Storage Fundamentals
• IOPS
• Throughput
• Latency
• SAN connectivity
• SAN Cache

4. IOPS
• # of distinct Input/Output operations (reads or writes) per second
• Several vendors advertise "just" higher IOPS, sort of meaning less without
other metrics like
• Latency
• Block Size - 2k, 4k, 8k, 16k, 32k...?
• Read write ratio

5. Latency
• A measure of the time required for a sub-system or a component in that sub-system to
process a single storage transaction or data request
• Latency matters!!
• Varies by block size

6. Latency
• DB level latency can be in sub ms still with spinning drives
Reference - http://carymillsap.blogspot.com/2009/02/dang-it-people-theyre-syscalls-not.html
• Wait event is SYSCALL
• Db file sequential read – pread call (linux)
• wait n. In an Oracle context, the approximate response time of what is usually a
single operating system call (syscall) executed by an Oracle kernel process.
• SAN Cache
• FS Cache
• Oracle data doesn’t show IO queue at OS level
• Oracle has no visibility if IO got serviced by FS cache, SAN cache or by spinning
drive

7. Throughput
• How much data can be transferred from one location to another in a
given amount of time
Or
• MegaBytes/Second
• Important for large scans/sequential reads

8. SAN connectivity fundamentals
• Zoning
• Port base
• WWN base

9. Does your Database need Flash Storage?
• Is your database IO sensitive?
• Does it need consistent and low latency?
• Does it currently have high latency and driving a lot of IO?
• Database server CPU starved?
• 3 offerings depending upon workload:
• Hybrid Storage with spinning drives with Cache (RAM or FLASH)
• All Flash arrays
• Exadata

10. AFA (All Flash Array) POC – Choose the right tool
Reference - https://www.pythian.com/blog/oracle-database-load-testing-tools-overview/
• Tools available:
• Orion (Oracle IO numbers)
• Winsat
• Iometer
• HammerDB
• Oracle RAT (Real Application Testing)
• LoadRunner
• SLOB
• Swingbench
• ..
• ..

11. • SLOB – For IOPS (To Test Physical IO)
• Swingbench
• Order Entry - introduces heavy contention on a small number of
tables and is designed to stress interconnects and memory
• Sales History - to test the performance of complicated queries when
run against large tables.
Tools we picked

12. • SLOB – For IOPS (To Test Physical IO)
• Swingbench
• Order Entry - introduces heavy contention on a small number of
tables and is designed to stress interconnects and memory
• Sales History - to test the performance of complicated queries when
run against large tables.
Tools we picked

13. • 2 Linux physical servers:
• Dell PowerEdge R630
– Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz (Haswell) – 20 Cores
– 378G RAM, RAM Speed - DDR4 384G (24x16g dimms) @1866@MHz
– HBA – QLogic QLE2662 16Gb FC Adapter
• What max throughput we can drive on each box – ~3200 MBPS
• 2 node RAC ASM 12.1 Database
Oracle Parameter Swingbench SLOB
SGA_TARGET 15G 2G
SGA_MAX_SIZE 15G 15G
PGA_AGGREGATE_TARGET 100G 100G
PGA_AGGREGATE_LIMIT 200G 200G
JOB_QUEUE_PROCESSES 80 80
PARALLEL_MAX_SERVERS 552 552
PARALLEL_MIN_SERVERS 288 288
REDO CONFIGURATION 10 Groups(min). 1GB each. 10 Groups(min). 1GB each.
TEMP TABLESPACE 200GB 200GB
Environment Details

14. SLOB
Reference- https://kevinclosson.net/slob/
• IO workload generation toolkit
• Supports following testing
• Oracle Logical Read scaling
• Physical random single block read (db file sequential read wait
event)
• Random single block writes (DBWR flushing capacity)
• Redo logging IO
• Best tool to test IO subsystem

15. SLOB
• Create ~500GB dataset (256 SLOB schemas each 2 GB)
• Ran several tests with different # of users - 2,4,8,16,32,64,92,128,160,192,224 and 256

16. Initial Setup
https://kevinclosson.net/2014/08/04/slob-deployment-a-picture-tutorial/
• Create a tablespace for SLOB
• Run setup.sh - To load SLOB schemas
• Execute runit.sh - to run real tests

17. Slob.conf
UPDATE_PCT= 0 to 50 -–Update Activity
RUN_TIME=1800 --seconds
WORK_LOOP=0
SCALE=256000 --Numbers of Blocks
WORK_UNIT=64
REDO_STRESS=HEAVY #LITE
LOAD_PARALLEL_DEGREE=4
# Settings for SQL*Net connectivity:
ADMIN_SQLNET_SERVICE=slob_poc5
SQLNET_SERVICE_BASE=slob_poc5
#SQLNET_SERVICE_MAX=2
#SYSDBA_PASSWD=xxxxx
SYSDBA_PASSWD=manager

18. SLOB Test Setup
oracle@hostname:poc51> srvctl status service -d poc5_wlk
Service slob_poc5 is running on instance(s) poc51,poc52
Service slob_poc51 is running on instance(s) poc51
Service slob_poc52 is running on instance(s) poc52
tnsnames.ora
slob_poc51 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST =xxxx)(PORT = 1521)) (CONNECT_DATA =
(SERVER = DEDICATED) (SERVICE_NAME = slob_poc51)))
slob_poc52 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = xxxx)(PORT = 1521)) (CONNECT_DATA =
(SERVER = DEDICATED) (SERVICE_NAME = slob_poc52)))
slob_poc5 = (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = xxxx)(PORT = 1521)) (CONNECT_DATA =
(SERVER = DEDICATED) (SERVICE_NAME = slob_poc5)))
Test it
sqlplus user1/user1@slob1 <<EOF
show parameter instance_name
EOF
sqlplus user1/user1@slob2 <<EOF
show parameter instance_name
EOF

19. Some underbar parameter
simple.ora
# The following are needed to force db file sequential read. Omit for direct path short
scans.
# If you would like to investigate a blend of short scans mixed with single-block
random I/O
# then comment out these three parameters :
_db_block_prefetch_limit = 0
_db_block_prefetch_quota = 0
_db_file_noncontig_mblock_read_count = 0

20. SLOB Caveat
• It overwrites logs
• Best for IOPS test but not good for driving high throughput

21. Runit wrapper script

22. Runit wrapper script
Better naming convention
Doesn’t overwrite files
Make Buffer Cache
small to stress IO
subsystem

23. IOPS/MBPS script
Number of Number of Total Number of Number of Total Total Total
DATE_TIME DATE_TIME Small reads Small writes Small IOPS large reads large writes Large IOPS MBPS read MBPS written Total MBPS Total IOPS
-------------------- -------------------- ------------ ------------- ----------- ------------ ------------- ----------- ---------- ------------- ---------- ----------
02/05/18_00_19_00_21 02/05/18_00_19_00_21 222 2716 2938 18 14 32 16 35 51 2970
02/05/18_00_21_00_51 02/05/18_00_21_00_51 70374 32138 102512 222 694 916 769 931 1700 103429
..
..
02/05/18_07_51_08_56 02/05/18_07_51_08_56 18 3 21 1738 10 1748 1728 2 1730 1769
Throughput = IOPS x I/O size
In what case IOPS
and Throughput
could be same?

24. Other tools – db_capacity.sql
• Leverages google chart
• Thanks to Mauro and Carlos for help 

25. Other tools – db_capacity.sql

26. Other tools – db_capacity.sql

27. Other tools – db_capacity.sql

28. Update% - 10 (To change read/write ratio)

29. Update% - 10 (To change read/write ratio)
0
1000
2000
3000
4000
2 4 8 16 32 64 96 128 224 256
# of SLOB Sessions
MBPS
MBPS
0
100000
200000
300000
400000
2 4 8 16 32 64 96 128 224 256
# of SLOB Sessions
IOPS
IOPS
0
1
2
3
4
5
2 4 8 16 32 64 96 128 224 256
ms
# of SLOB Sessions
Single Block Read Latency (ms)
Avg
Max
0
2
4
6
8
2 4 8 16 32 64 96 128 224 256
ms # of SLOB Sessions
log file parallel write (ms)
Avg
Max

30. Update% - 10 (To change read/write ratio)
27082
49337
90660
163114
292842
347478 345245
323205 326606 325784
0.39 0.43 0.47 0.51 0.56
1.02
1.57
2.23
3.62
4.04
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
0
50000
100000
150000
200000
250000
300000
350000
400000
2 4 8 16 32 64 96 128 224 256
Rd
RdLatency(ms)
IOPS
# of SLOB Sessions IOPS Rd Latency(ms)

31. Swingbench testing (Throughput)
 Shwizard to build Sales History Schema
 500GB schema was created (with range partition)
 Several tests (SH) were run with different parallel/Index settings
 Swingbench config info
• NumberOfUsers 16
• MinDelay 0
• MaxDelay 0
• InterMinDelay 1000 ms
• InterMaxDelay 2000 ms
• RunTime 1h:5 min
• StatsCollectionStart 0:4 min
• StatsCollectionEnd 1h:4 min
http://dominicgiles.com/swingbench.html

32. Xml config file
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<SwingBenchConfiguration xmlns="http://www.dominicgiles.com/swingbench/config">
<Name>Sales History</Name>
<Comment>Configuration file for the Sales History Benchmark</Comment>
<Connection>
<UserName>sh_500</UserName>
<Password>sh_500</Password>
<ConnectString>//eihtlrac-scan/slob_poc5</ConnectString>
<DriverType>Oracle jdbc Driver</DriverType>
<Properties>
<Property Key="FetchSize">20</Property>
<Property Key="StatementCaching">50</Property>
<Property Key="TcpNoDelay">true</Property>
</Properties>
<InitilizationCommands>
<Command>alter session set sql_trace=false</Command>
</InitilizationCommands>
</Connection>
<Load>
<NumberOfUsers>16</NumberOfUsers>
<MinDelay>0</MinDelay>
<MaxDelay>0</MaxDelay>
<InterMinDelay>1000</InterMinDelay>
<InterMaxDelay>2000</InterMaxDelay>
<QueryTimeout>55</QueryTimeout>
<MaxTransactions>-1</MaxTransactions>
<RunTime>1:5</RunTime>

33. Charbench
DIR='/u01/swingbench'
BDT=$(date +%Y%m%d)
BTM=$(date +%H%M%S)
$DIR/bin/charbench -c $DIR/configs/ei_sh.xml -r ${FILE_NM}_results_${BDT}_${BTM}.xml

34. Output xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<Results xmlns="http://www.dominicgiles.com/swingbench">
<Overview>
<BenchmarkName>Sales History</BenchmarkName>
<Comment>Configuration file for the Sales History Benchmark</Comment>
<TimeOfRun>Feb 7, 2018 9:47:44 PM</TimeOfRun>
<TotalRunTime>1:00:00</TotalRunTime>
<TotalLogonTime>0:00:00</TotalLogonTime>
<TotalCompletedTransactions>40</TotalCompletedTransactions>
<TotalFailedTransactions>1</TotalFailedTransactions>
<AverageTransactionsPerSecond>0.01</AverageTransactionsPerSecond>
<MaximumTransactionRate>5</MaximumTransactionRate>
<AverageResponse>1300652.25</AverageResponse>
<MinimumResponse>632</MinimumResponse>
<MaximumResponse>3102874</MaximumResponse>

35. Test Summary

36. Vendor rating
• Support
• Security
• Reporting
• Reliability
• Performance
• Management
• Functionality
• Cost
• Storage efficiency – (Dedup, compression)
• Scalability

37. Appendix - A
Test case – "Spectre" and "Meltdown" vulnerability patching with BIOS update

38. Appendix-B db_capacity.sql

39. Appendix-C Stack Ranking of Oracle databases

40. Stack rank your databases
Database Rank IOPS GB/s Latency Cores Storage
ORAprod 1 1.5M 100 <1ms 128 100TB
ORAprod 2 1.4M 47 <1ms 128 75TB
DB2prod 3 1.3M 46 <1ms 128 75TB
ORAprod . . . . . .
DB2prod . . . . . .
SQLprod . . . . . .
ORAprod . . . . . .
DB2prod . . . . . .
SQLprod . . . . . .
ORAtest . . . . . .
DB2test . . . . . .
SQLtest . . . . . .
ORAdev . . . . . .
SQLprod . . . . . .
SQLtest . . . . . .
SQLdev 3000+ 100 1 >10ms 2 20GB
High End
Physical
High End
Virtual
Virtual
On-Prem
As-A-Service
Off-Prem
As-A-Service
80%
virtual
start
here

41. Enterprise wide 95th, 98th and 99th database level stats

42. References:
https://kevinclosson.net/slob/
https://kevinclosson.net/2014/08/04/slob-deployment-a-picture-tutorial/
http://dominicgiles.com/swingbench.html
https://www.pythian.com/blog/oracle-database-load-testing-tools-overview/
https://flashdba.com/

43. Q&A