Make Oracle scream with Flash
storage
A bit about myself
Eyal Markovich
Senior Director of Performance

• 20 years APM and database performance experience
• 18 years in Precise working with hundreds of
customers
• 4 years in Kaminario managing the performance
team
Performance Analyses Learnings:
1. Business-critical applications require flash
2. Not all flash products are created equal
3. They Require 3D performance consistency
More than 80% of the
performance issues are traced
back to poorly performing
storage
Let’s start with an example

• Large financial institute
• Slow Batch job run for 7 hours
• Often exceeding batch window
• Critical data not available for the business
20 years: similar disk technology

Selecting 1% of the database …
Performance:
5X
1990 1994 1998 2003 2011
Data:
1400X

1
Min

2
Min

3.5
Min

18
Min

11.5
Hours

1.4 GB

4 GB

9 GB

73 GB

2 TB
Storage Architecture Evolution

XIV

- No shared
storage

- Over provisions
$$$
- No shared
workloads

- Difficult to grow

- Can’t scale easily

- Under utilized
capacity

- Creates silos
- Hard to manage

-Disrupted
Performance
Limited
-Doesn’t fit flash
-Performance
Limited

- Same as DAS
- Tuned for single
workloads

- Real Time,
elastic and agile
- 3D performance

- IOPS focused only

- Resilient

- Inconsistent

- Scalable

- Not resilient

- Best TCO

- Not scalable
Flash – Competitive Landscape
3D Performance

Silo’d ($$$ & Resources)
Limited Workloads
Limited SW stack or 3rd party
Limited Scalability
No enterprise HA

Silo’d ($$$ & Resources)
One Workload
No SW stack
No Scalability
No enterprise HA

Mixed Workloads
Full SW stack
Easiest management
Best TCO
Enterprise HA

Silo’d ($$$ & Resources)
Limited Workloads
Limited Scalability
SLA under failure

Software
Capabilities
Back to the example

• Large financial institute
• Slow Batch job run for 7 hours
• Often exceed batch window
• Critical data not available for the business

I/O Wait before
Kaminario

I/O Wait
After Kaminario

CPU
What about my Oracle instance?

• Do I need Flash for my Oracle?
CPU

Lock Net

I/O Waits

8:00 am

• What type of Flash is best for me?
• SSD cards? Hybrid array?
• SAN flash array? Scale out SSD flash array?

9:00 am
What about my Oracle instance?

• Do I need Flash for my Oracle?
What will be the effect of Flash on my workload?
What will be the effect of Flash on my workload?

db file sequential read
What will be the effect of Flash on my workload?

db file scattered read
What will be the effect of Flash on my workload?

Direct Path Read
Sample SQL Query:
Select /*+ Parallel(emp 4)
*/ * from Employee emp;
Execution Plan:
SELECT STATEMENT
PX COORDINATOR
PX RECEIVE
PX SEND RANGE
PX BLOCK ITERATOR
TABLE ACCESS FULL
EMPLOYEE
What will be the effect of Flash on my workload?

Log file sync wait
What will be the effect of Flash on my workload?

Direct path read temp and direct path write temp
Performance Analysis Issues:

•
•
•
•
•
•
•

Random read latency for small block size
Large latency for sequential reads
Mixed workload applications running both OLTP & reports
Parallelism set too high or too low
Writes (DML) affecting read latencies
Storage unable to support temporary workload
Large latency for writes

Inconsistent Performance Across Workloads
SAP Example: Simple Workload

Two Distinct Workloads:
1.
2.

Dialog User activity
Programs (batch)

Running Overnight
• Large sequential scans
• High parallelism on the database
• High throughput
SAP Example: Simple Workload

What we need from storage to improve the
performance:
LOW
Throughput
1,483 MB/s

HIGH

IOPS
4,410

Latency
0.89 ms
SAP Example: Complex Workload

 Can you run the SAP programs during the day?
Activity:
• What will the effect be on User activity?
• How well will the SAP programs perform?
It’s All About Storage:
• What will the effect be on storage latencies?
• What will the effect be on storage throughput?
• Will storage performance remain consistent ?

Can your system tolerate mixed workload?
SAP Example: Complex Workload

You need flash that supports mixed workloads:
•
•

Consistently low latencies regardless of the throughput
Delivering high throughput regardless of the IOPS

LOW
Throughput
1,713 MB/s

HIGH

IOPS
24,426

Latency
.56 ms
What are Kaminario K2 differentiators?

Storage done better
Scale-Out

3D Performance

 Granular & Linear (6120 TB)

 Consistent

 Automatic
performance &
endurance

 Linear scale (200K- 2M+
IOPS, 3-30 GB/s)

 7 year flash endurance
 Simple & Intuitive
Global management
 Low TCO than
alternatives

 Lowest Latency (280 us)

 Mixed workload
concurrently
 Minimal performance
degradation (<25%)
during recovery

Safe & Reliable
 No Single Point of
Failure
 Self healing – N+1
 Instantaneous, space
efficient, snapshots
 Non disruptive
upgrades

 Full node recovery
~ 90 minutes
Kaminario K2 Building Block

Enterprise Grade Highly Available Building Block
Cluster together to scale-out Performance or Capacity
Feature
Flash

K-Block
K-Node

Specification
8 MLC Flash (800 GB or 1.6 TB) per K-Node

Usable Capacity

3 TB / 6 TB per K-Node
(6 – 12 TB) or (12- 24 TB) per K-Block

Performance

IOPS - 200K-410K
Throughput - 3–6GB/s
Latency - 280 us

Footprint

6 RU – 8 RU

Power

Typical 1.4kW-2kW

High Availability

System components fully redundant
Dual Processors, dual battery

Host Ports

2 host ports per K-Node
8 x 8Gbp FC HBA or 8 x 10GbE iSCSI per K-Block

K-Node
K-Node
K-Node
Customers Span Industries & Use Cases
SPEAR Technology - Scalability
Flash to the Rescue

Commodity HW.
It’s all about SW architecture
SPC-1 World Record Performance

(.80 cents per SPC-1 IOPS)
SPC-1 2013 Configuration
CONFIGURATION

•
•
•
•

86 TB System
7 K-Blocks
34U Footprint
SMART – All Flash

CONFIGURATION RESULTS

• 3 Million Read IOPS at under 2ms latency
•

2.8 Million Read IOPS at under 1ms

•

1.2 Million Write IOPS at under 1ms

• 56 GB/s Read/Write Throughput
3D Performance: Latency under failure
Kaminario Scale-Out Flash Array
Hybrid & Monolithic Flash
Get a Performance Assessment
www.kaminario.com/performance

Make Oracle scream with Flash Storage - Kaminario

  • 1.
    Make Oracle screamwith Flash storage
  • 2.
    A bit aboutmyself Eyal Markovich Senior Director of Performance • 20 years APM and database performance experience • 18 years in Precise working with hundreds of customers • 4 years in Kaminario managing the performance team
  • 3.
    Performance Analyses Learnings: 1.Business-critical applications require flash 2. Not all flash products are created equal 3. They Require 3D performance consistency
  • 4.
    More than 80%of the performance issues are traced back to poorly performing storage
  • 5.
    Let’s start withan example • Large financial institute • Slow Batch job run for 7 hours • Often exceeding batch window • Critical data not available for the business
  • 6.
    20 years: similardisk technology Selecting 1% of the database … Performance: 5X 1990 1994 1998 2003 2011 Data: 1400X 1 Min 2 Min 3.5 Min 18 Min 11.5 Hours 1.4 GB 4 GB 9 GB 73 GB 2 TB
  • 7.
    Storage Architecture Evolution XIV -No shared storage - Over provisions $$$ - No shared workloads - Difficult to grow - Can’t scale easily - Under utilized capacity - Creates silos - Hard to manage -Disrupted Performance Limited -Doesn’t fit flash -Performance Limited - Same as DAS - Tuned for single workloads - Real Time, elastic and agile - 3D performance - IOPS focused only - Resilient - Inconsistent - Scalable - Not resilient - Best TCO - Not scalable
  • 8.
    Flash – CompetitiveLandscape 3D Performance Silo’d ($$$ & Resources) Limited Workloads Limited SW stack or 3rd party Limited Scalability No enterprise HA Silo’d ($$$ & Resources) One Workload No SW stack No Scalability No enterprise HA Mixed Workloads Full SW stack Easiest management Best TCO Enterprise HA Silo’d ($$$ & Resources) Limited Workloads Limited Scalability SLA under failure Software Capabilities
  • 9.
    Back to theexample • Large financial institute • Slow Batch job run for 7 hours • Often exceed batch window • Critical data not available for the business I/O Wait before Kaminario I/O Wait After Kaminario CPU
  • 10.
    What about myOracle instance? • Do I need Flash for my Oracle? CPU Lock Net I/O Waits 8:00 am • What type of Flash is best for me? • SSD cards? Hybrid array? • SAN flash array? Scale out SSD flash array? 9:00 am
  • 11.
    What about myOracle instance? • Do I need Flash for my Oracle?
  • 12.
    What will bethe effect of Flash on my workload?
  • 13.
    What will bethe effect of Flash on my workload? db file sequential read
  • 14.
    What will bethe effect of Flash on my workload? db file scattered read
  • 15.
    What will bethe effect of Flash on my workload? Direct Path Read Sample SQL Query: Select /*+ Parallel(emp 4) */ * from Employee emp; Execution Plan: SELECT STATEMENT PX COORDINATOR PX RECEIVE PX SEND RANGE PX BLOCK ITERATOR TABLE ACCESS FULL EMPLOYEE
  • 16.
    What will bethe effect of Flash on my workload? Log file sync wait
  • 17.
    What will bethe effect of Flash on my workload? Direct path read temp and direct path write temp
  • 18.
    Performance Analysis Issues: • • • • • • • Randomread latency for small block size Large latency for sequential reads Mixed workload applications running both OLTP & reports Parallelism set too high or too low Writes (DML) affecting read latencies Storage unable to support temporary workload Large latency for writes Inconsistent Performance Across Workloads
  • 19.
    SAP Example: SimpleWorkload Two Distinct Workloads: 1. 2. Dialog User activity Programs (batch) Running Overnight • Large sequential scans • High parallelism on the database • High throughput
  • 20.
    SAP Example: SimpleWorkload What we need from storage to improve the performance: LOW Throughput 1,483 MB/s HIGH IOPS 4,410 Latency 0.89 ms
  • 21.
    SAP Example: ComplexWorkload  Can you run the SAP programs during the day? Activity: • What will the effect be on User activity? • How well will the SAP programs perform? It’s All About Storage: • What will the effect be on storage latencies? • What will the effect be on storage throughput? • Will storage performance remain consistent ? Can your system tolerate mixed workload?
  • 22.
    SAP Example: ComplexWorkload You need flash that supports mixed workloads: • • Consistently low latencies regardless of the throughput Delivering high throughput regardless of the IOPS LOW Throughput 1,713 MB/s HIGH IOPS 24,426 Latency .56 ms
  • 23.
    What are KaminarioK2 differentiators? Storage done better Scale-Out 3D Performance  Granular & Linear (6120 TB)  Consistent  Automatic performance & endurance  Linear scale (200K- 2M+ IOPS, 3-30 GB/s)  7 year flash endurance  Simple & Intuitive Global management  Low TCO than alternatives  Lowest Latency (280 us)  Mixed workload concurrently  Minimal performance degradation (<25%) during recovery Safe & Reliable  No Single Point of Failure  Self healing – N+1  Instantaneous, space efficient, snapshots  Non disruptive upgrades  Full node recovery ~ 90 minutes
  • 24.
    Kaminario K2 BuildingBlock Enterprise Grade Highly Available Building Block Cluster together to scale-out Performance or Capacity Feature Flash K-Block K-Node Specification 8 MLC Flash (800 GB or 1.6 TB) per K-Node Usable Capacity 3 TB / 6 TB per K-Node (6 – 12 TB) or (12- 24 TB) per K-Block Performance IOPS - 200K-410K Throughput - 3–6GB/s Latency - 280 us Footprint 6 RU – 8 RU Power Typical 1.4kW-2kW High Availability System components fully redundant Dual Processors, dual battery Host Ports 2 host ports per K-Node 8 x 8Gbp FC HBA or 8 x 10GbE iSCSI per K-Block K-Node K-Node K-Node
  • 25.
  • 26.
    SPEAR Technology -Scalability
  • 27.
    Flash to theRescue Commodity HW. It’s all about SW architecture
  • 28.
    SPC-1 World RecordPerformance (.80 cents per SPC-1 IOPS)
  • 29.
    SPC-1 2013 Configuration CONFIGURATION • • • • 86TB System 7 K-Blocks 34U Footprint SMART – All Flash CONFIGURATION RESULTS • 3 Million Read IOPS at under 2ms latency • 2.8 Million Read IOPS at under 1ms • 1.2 Million Write IOPS at under 1ms • 56 GB/s Read/Write Throughput
  • 30.
    3D Performance: Latencyunder failure Kaminario Scale-Out Flash Array Hybrid & Monolithic Flash
  • 31.
    Get a PerformanceAssessment www.kaminario.com/performance

Editor's Notes

  • #5 Additional info we’ve found when doing these performance assessments
  • #24 SPEAR minimizes writes to the Flash media and maximizes Flash endurance.It utilizes a system-wide write cache that eliminates hotspots, and it distributes writes across all Flash media in the system, thus sustaining global wear leveling. SPEAR’s efficient management of metadata eliminates writes to the Flash during updates and facilitates space efficient snapshots that do not incur additional writes. K2 deploys enterprise-grade level SSDs with advanced Flash management capabilities that optimize the endurance of theFlash . With SPEAR’s abilities, Kaminario has a warranty of 7 years for Flash enduranceWe use enterprise SSDs with advance error correction techniques and with optimization related to the way the data is written to the Flash drives.
  • #25 The K-Node is build from 4 1U K-Servers. Super Micro ServersThis unit is best optimized for performance, and the performance is in the ball park of these figures.Now, you just need so scale that performance and capacitySpeak verbally performance enhancements % improvement vs. V3 . Even though cost effective Enterprise SAS MLC – IOPS and Throughput much better. Latency bit lower but still within &lt;0.5 ms target. Also talk about performance during recovery – how using backup to Flash meets customer need for performance even during worst times!The K-Node is build from 4 1U IO serversThis unit isbest optimized for performance, and these are the figures.Now, you just need so scale that performance and capacityLet’s talk about V4’s HW architecture.We use a 1U rack mount architecture, which allows us to benefit from good footprint and power, as well as high media density.The servers are based on the latest and best processing power available today, and our advantage here is that we can easily benefit from any progress made in this area.Since we are not bound to a single vendor, we can always benefit.As said before, the K2 is an all SSD array, where the primary and secondary media are MLC based SAS Solid State Drives.Our system is vendor agnostic, and we are implementing a dual vendor strategy. One of our vendors is SMART. (SECOND VENDOR: if asked, we are still in the works of deciding on a second vendor)(NAND: if asked, the NAND supplier of the drives we are utilizing is Toshiba)The spare node, of our N+1 design, is consolidated together with the management servers, which improves foot print, power and cost.The spare is used only in the scenario of a recovery, and IOs are always served from an SSD device.We are currently deploying 8 drives of 800GByte per 1U, with the intention of deploying 1.6TByte drives per 1U.Also here, we can benefit from market changes.(SHAI: the MLC is a 2X technology, 24nm, 10 writes per day per block, full capacity for 5 years) Optimus™ SAS SSD There is no UPS in the system, rather each server has a battery that insures a graceful shutdown upon a power failure, securing the data.Our network backbone is a fully redundant 10GigaBitEtherent and we support FibreChannel and iSCSI protocols.The K2 is managed from the centralized K-MGMT block, no matter how you scale.It contains two redundant MGMT Servers, and two redundant 10GigaBitEthernet switches.It is in charge of managing all aspects of K2: create volumes, monitoring, external connections and more.EXPLAIN: Network usage for internal traffic onlyDataControl (MGMT)
  • #31 Take the lower red to just below 2 and the upper line to 20 ishAdd performance logo guaranteed5X and goes to 10-15x