Presentation oracle super cluster t5-8 technical deep dive

Download this slide
http://ouo.io/WORUjg

Oracle SuperCluster T5-8
Technical Deep Dive
Insert Presenter’s Name Here
Insert Presenter’s Title Here
*Oracle Internal Use Only*

The following is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated
into any contract. It is not a commitment to deliver any material, code,
or functionality, and should not be relied upon in making purchasing
decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole
discretion of Oracle.

Agenda
 Oracle SuperCluster Overview
 Product Specifications
 Use Cases and Competitive Analysis

Engineered Systems for
Oracle Database
Oracle
SuperCluster
T5-8
Oracle
Exadata
X4-2
Oracle
Exadata
X3-8
Oracle
SuperCluster
M6-32
NEW!

Months
PerformanceAchievement
Assemble
dozens of
components
Measure,
diagnose, tune
and reconfigure
Measure,
diagnose, tune
and reconfigure
Live to
production
Multi-vendor
testing
Test and debug
production
failure modes
Oracle SuperCluster
Days
PerformanceAchievement
Eliminate the Delays of CustomAssembly
Build-It-Yourself Approach
Consolidation Accelerated

Time to Value
Than Standalone
Components for
Service Delivery
Readiness
0
50
100
150
200
250
300
Oracle SuperCluster Do It Yourself
Infrastructure Installation and Configuration
Software Installation and Configuration
Consolidate Faster
Deploy Applications In Less Time - and See the Results Sooner
Hours
5X
FASTER

Consolidate More
Slash Storage Requirements with Database Compression
10X
 SuperCluster’s Exadata and ZFS
storage uses Hybrid Columnar
Compression (HCC) for up 10X data
compression
 Consolidate more now and scale more
efficiently
 Reduces the need for additional SAN-
based storage

Engineered Systems for
Oracle Database
Oracle
SuperCluster
T5-8
Oracle
Exadata
X4-2
Oracle
Exadata
X3-8
Oracle
SuperCluster
M6-32
NEW!

Engineered Systems for Oracle Database
Incredible Power, Flexibility and Choice
Exadata
X4-2
Exadata
X3-8
SuperCluster
T5-8
SuperCluster
M6-32
OLTP, Data Warehouse & Consolidation Excellent Excellent Excellent Excellent
In-memory Workloads Good Very Good Very Good Unbeatable
Virtualization Supported No No Yes Yes
Database Versions Supported 11g/12c 11g/12c 9i/10g/11g/12c 9i/10g/11g/12c
Also Runs Applications (Optional) No No Yes Yes
Sub-capacity Software Licensing No No Yes Yes
Oracle Solaris Support Yes No Yes Yes
Oracle Linux Support Yes Yes No No

Introducing Oracle SuperCluster T5-8
Oracle’s Fastest Engineered System
EXTREME PERFORMANCE
 Powered by the world’s fastest database and application server
 10X faster database and Java applications
HIGHEST EFFICIENCY
 10X database compression
 Zero overhead virtualization
NO SINGLE POINT OF FAILURE
 Zero downtime maintenance

Oracle SuperCluster
Best for Oracle. Runs Your Existing Workloads
SPARC T5 Compute Pool
17 World Records over IBM and HP
across every tier
Exadata Storage Servers
750K IOPS, 32 GB/s query
throughput
Exalogic Elastic Cloud
10x Java performance
Integrated ZFS Storage
2x faster and 2x better price
performance than NetApp
Oracle Solaris 11
Cloud provisioning in seconds
Unmatched Scalability
Virtualization
Oracle VM Server for SPARC
No cost and zero overhead
InfiniBand
5-8x the speed of current networks
Enterprise Manager Single
pane of glass to monitor and
manage entire system

Why SuperCluster Over Regular Cluster?
• Building a cluster ad-hoc is complex
- Proper redundant network and storage connectivity
- Setup of the OS
- Setup of the cluster software
- Install and setup of application agents
- Integration with enterprise management framework
- Testing to make sure it works (i.e. find out what step was skipped….)
• Getting support
- Root-cause analysis requires working with individual support groups (i.e.
server group, network group, storage group, software group)
- Time to resolution is critical!

Agenda

Configuring Oracle’s Oracle SuperCluster
Introduction to Configurations
• Oracle SuperCluster is configured according to your requirements
– SPARC T5-8 compute nodes are split into “domains”
– Based on Oracle VM Server for SPARC (LDoms)
• Supported Domain Types
– Database Domain (Oracle Database 11gR2 today, 12c later)
– Application Domains (Oracle Solaris 11 or Oracle Solaris 10)
• SPARC T5-8 Nodes
– Half rack has two half-populated nodes
– Full rack has two fully-populated nodes
– Nodes can be configured independently

The New Oracle SuperCluster T5-8
Complete Architecture Deliver Extreme Performance
InfiniBand Switches
ZFS Storage Appliance
SPARC T5-8 Compute Nodes
• 2,272 CPU threads
• 4.7 TB DRAM
• 356 TB Hard Disk
• 16.8 TB Flash
• 1.2M IOPS
• 62 GB/sec Storage Bandwidth
• 960Gb/sec InfiniBand Interconnect

Oracle SuperCluster T5-8
Half Rack
- 2 x SPARC T5-8 compute nodes
 4 x T5 Processors @ 3.6 GHz
 1 TB of memory (64 x 16GB)
 8 x 900 GB internal drive (SAS)
 4 x Sun Dual 10 GbE SFP+ PCIe 2.0 Low Profile Adapter
(2 port)
 4 x Sun Quad Data Rate (QDR) InfiniBand PCIe Low
Profile HCA (2 port)
– 4 x Exadata Storage Servers, either
 High Performance (28.8 TB capacity total)
 High Capacity (144 TB capacity total)
- Oracle ZFS Storage ZS3-ES array
- 3 x Sun Data Center InfiniBand Switch 36
- 1 x GbE Management Switch
Full Rack
- 2 x SPARC T5-8 compute nodes
• 8 x T5 Processors @ 3.6 GHz
• 2 TB of memory (128 x 16GB)
• 8 x 900 GB internal drive (SAS)
• 8 x Sun Dual 10 GbE SFP+ PCIe 2.0 Low
Profile Adapter (2 port)
• 8 x Sun Quad Data Rate (QDR) InfiniBand
PCIe Low Profile HCA (2 port)
- 8 x Exadata Storage Servers, either
• High Performance (57.6 TB capacity total)
• High Capacity (288 TB capacity total)
- Oracle ZFS Storage ZS3-ES array
- 3 x Sun Data Center InfiniBand Switch 36
- 1 x GbE Management Switch
The half rack configuration can be upgraded to a full rack configuration!

SPARC T5-8 Server
The Industry’s First Multithreaded Enterprise Datacenter Server for
Mission Critical Workloads
 Compute
– 4x or 8x SPARC T5 16-core CPU (1024 threads max)
– 64x or 128x 16 GB DDR3 DIMMs
 I/O and Storage
– 16x PCIe 3.0 x8 slots, with PCIe carriers
– 4x 10GbE ports built-in for management
– 8x or 16x 10GbE ports for data access
– 8x 2.5” SAS 3.0 drives
 Availability and Management
– RAID 0/1, Hot-swap fans & PSUs
– Oracle Integrated Lights Out Manager Service Processor
– Oracle Enterprise Manager Ops Center

SPARC T5 Processor
• Features
• 16 S3 cores, 16-128 Strands @
3.6Ghz
• Single or multi-threaded operation per
core
• System scalability to 8 sockets
• SPARC Core S3
• 1-8 Strand Dynamically Threaded
Pipeline
• ISA-based Crypto-acceleration
• 8MB Shared L3$
• Integrated I/O
• Double I/O bandwidth over T5
• 2 x8 Lane PCIe 3.0 @ 8GT/s
• System Scalability
• 7 Coherence Ports for scalability to
8S
• Power Management
• Dynamic Voltage Frequency Scaling
• Downclock, Overclock

SPARC T5 CPU Block Diagram
2x8PCIe3.0@8GBps
16GBpseachdirection
8 threads per Core
IO
Subsystem
BoB BoB BoBBoB BoB BoB BoB
DDR3 – 1066 MHz
Memory
Control
Coherency4x4Switch
Memory
Control
Memory
Control
Memory
Control
8 x 9 Crossbar (~1TBps bandwidth)
C3 C4 C5 C6 C7 C8 C9 C10 C11C0 C1 C2 C12 C13 C14 C15
L3$ B0L3$ B0
1MB,1
6-way
L3$ B0
1MB,1
6-way
L3$ B2
1MB,1
6-way
L3$ B1
1MB,1
6-way
L3$ B3
1MB,1
6-way
L3$ B0L3$ B0
1MB,1
6-way
L3$ B4
1MB,1
6-way
L3$ B6
1MB,1
6-way
L3$ B5
1MB,1
6-way
L3$ B7
1MB,1
6-way
128 KB L2$
16 KB L1I$
16 KB L1D$
FGU
Crypto
SPARC S3
Core
BoB
DDR3 – 1066 MHz DDR3 – 1066 MHz DDR3 – 1066 MHz
Coherence Unit Coherence Unit Coherence Unit Coherence Unit Link 0
Link 1
Link 2
Link 3
Link 4
Link 5
Link 6
Coherency Links
12.8 Gbps per lane
- 12 lanes per link

Critical Thread
Priority Performance On Demand
Critical thread gets all core resources.
The Results
 2x speed up for SOA startup
 15% improvement in database response time
 20% improvement for Coherence
No
Critical
Thread
Important thread competes
for core resources.

PCIe Data Paths: Half Rack
 Two root complexes per T5
processor
 RCs must be assigned in pairs,
from the same T5
 Each PCIe port on a T5
processor controls a two PCIe
slots
 This provides for bare metal I/O
performance
c e e 4 8 8a
c

PCIe Data Paths: Full Rack
 Two root complexes per T5
processor
 RCs must be assigned in pairs,
from the same T5
 Each PCIe port on a T5
processor controls a single
PCIe slot
 I/O is not virtualized
 This provides for bare metal I/O
performance

Root Complex
 Each processor’s root
complexes must be assigned
to the same domain
 Example:
/pci@300/pci@1/pci@0/pci@6 &
/pci@340/pci@1/pci@0/pci@6
from CPU0 are assigned to the
same logical domain
PM CPU Switch I/O Slot Root Complex Path
0 0 0 1 /pci@300/pci@1/pci@0/pci@6
0 0 1 3 /pci@340/pci@1/pci@0/pci@6
0 1 2 9 /pci@380/pci@1/pci@0/pci@a
0 1 3 11 /pci@3c0/pci@1/pci@0/pci@e
1 2 0 2 /pci@400/pci@1/pci@0/pci@c
1 2 1 4 /pci@440/pci@1/pci@0/pci@c
1 3 2 10 /pci@480/pci@1/pci@0/pci@4
1 3 3 12 /pci@4c0/pci@1/pci@0/pci@8
2 4 1 5 /pci@500/pci@1/pci@0/pci@e
2 4 2 7 /pci@540/pci@1/pci@0/pci@e
2 5 3 13 /pci@580/pci@1/pci@0/pci@a
2 5 4 15 /pci@5c0/pci@1/pci@0/pci@8
3 6 1 6 /pci@600/pci@1/pci@0/pci@8
3 6 2 8 /pci@640/pci@1/pci@0/pci@8
3 7 3 14 /pci@680/pci@1/pci@0/pci@4
3 7 4 16 /pci@6c0/pci@1/pci@0/pci@4

 Exadata Storage Servers run Oracle Enterprise Linux and Exadata Storage software. They currently
consist of the following hardware:
– X4-2L with:
 2 sockets Xeon E5-2630 6 cores @ 2.6GHz 15MB L3
 96GB Memory (4 * 8GB + 4 * 16GB DDR3 1600MHz LV DIMMs)
 SAS-2 RAID HBA
 12 disks, either high performance or high capacity
• High Performance: 1.2TB@10Krpm SAS2 SSF 2.5" Drives (total 14.4TB)
• High Capacity: 4TB@7200rpm SAS2 3.5" Drives (total 48TB)
• 4 x 800 GB Sun FlashAccelerator F80 PCIe cards (total 3.2TB)
• 2 InfiniBand 4X QDR (40Gb/s) Ports (PCIe 3.0)
 They can only be used by Database Domains
 Four are installed in the half rack config; eight are installed in the full rack config
They are the EXACT same storage servers as the Exadata Storage Server X4-2L

Key Exadata Innovations
+ ++
• Hybrid Columnar Compression
– 10x compression for warehouses
– 15x compression for archives
Data remains
compressed
for scans
and in Flash
Space Savings
Cascade to
Copies
uncompressed
compress
primary DB
standby
test
dev
backup
• Smart Scale-Out Storage
– InfiniBand connected servers
– Smart Scan query offload
• Smart PCI Flash Cache
– Transparent cache in front of disk
– Accelerates random I/O up to 30x
– Quadruples data scan rate

Improved Exadata Storage Server X4-2L
Enhanced Smart Flash Cache
• New software caches write I/Os in flash memory
−20x write improvement
−10x write improvement from previous storage servers
• Cache is transparent, persistent, and fully redundant

Improved Exadata Storage Server X4-2L
Enhanced X3 Database Machine Over X3
All X4 Storage Servers
2X Larger Physical Flash Memory
Up to 4x Larger Logical Flash Memory
86% More IOPs
33% Larger High Capacity Disks
2X Larger High Performance Disks
Smart Flash Cache Compression
2.8M Flash Reads running SQL
4TB Disks, 672 TB per Rack
1.2 TB SAS 2.5” Disks (Still 12 per Cell)
44 TB of Flash Memory per Rack

Exadata Smart Flash Log
Accelerate Transaction Response Times using Flash
 Uses Flash for Database Logs in a clever way
– Flash is fast but has slow outliers
– Erase cycles, wear leveling, etc.
 Smart Flash Log feature transparently uses Flash as a parallel write
cache to disk controller cache
– Whichever write completes first wins (disk or flash)
 Better response time and more throughput
 Uses almost no flash capacity (0.1% of capacity)
Default (on left)
- Choppy response
- High Outliers
Smart Flash Log
- 3x faster response
- Much lower outliers
Transaction Response Times
Smart Flash Log Enabled
Automatic and
Transparent

Oracle ZFS Storage ZS3-ES
Shared Storage
 Provides shared 80TB of storage for all Logical Domains
– Not mounted by default in the Database Domain.Active mounts
are established to all other Domains.
 Data can only be accessed in the IB backbone. External access is via built-in GbE ports used
exclusively for DR and data replication.
 Two controllers for high-availability and Infiniband
 292 GB of integrated Flash
 Stores quorum device for Solaris Cluster
 An additional IB-enabled ZFS SA device can be attached with IB ports connected to both IB Leaf
switches
 Accessible from Database Domain after manual mount
– Can be used for RMAN backups or flat file staging

Disk
Controllers
Disk Trays
 ZS3-ES Clustered Storage Array, per
controller (2 controllers)
– 2 * SNB-EP E5-2658 (8-core, 2.1GHz CPU)
– 16 * 16GB DDR3 1600MHz DIMMS (256GB
total)
– 1 * Dual-port InfiniBand HCA (CX2)
– 2 * 2.5" 900GB disks
– 2 * 1.6TB Readzillas (read-optimized SSDs,
3.2TB total)
– 1 * SAS HBA
– 1 * V2 PCIe LP Cluster card
– 2 * dual-port 10Gb Cu (4 ports total)
 1 Disk Shelf
– 20 * 4TB SAS disks @ 7200rpm
– 4 * 73GB Logzillas (write-optimized SSDs)

ZS3-ES Controller Rear Chassis View
ZS3-ES Controller Front Chassis View

Features Comparison
ZS3-ES 7320
CPU 2x 8-core 2.1 GHz X3-2 E5-2658 (Sandy Bridge) 2x 4-core 2.4GHz X2-2 E5620 (Westmere)
QPI Links 2 QPI Links. 32GB/s per QPI 1 QPI Link. 25.6GB/s
Memory 16 x 16GB (256GB) DDR3 1600MHz 96GB
Relative SpecINT 2.23x 1x
Cluster Yes. Yes.
Readzilla 2 x 1.6TB Sandisk SAS SSD 4x 512GB SATA SSD
NIC 2x embedded dual 10GbE 4 x 1GbE or 2 x 10GbE
External HBA SAS-2 SAS-2
Max Enclosures 1x DE2-24C or 2x DE2-24P 4x J4410

Other Storage Options
 Additional external storage options are supported for Oracle database releases prior
to 11gR2
– They are supported only when run in Application Domains
 Additional supported storage options are:
- SAN storage
- NAS storage, accessed using 10GbE
- An additional InfiniBand-enabled ZFS Storage Appliance
 Any ZFSSA can be added for non-11gR2 DB storage, applications, and Backup and Recovery
 Asingle ZFS Storage Appliance can be connected to any SuperCluster via InfiniBand without
requiring additional InfiniBand switch hardware
 Best practice performance recommendations based on NFS over IPoIB are available
for databases running on ZFS Storage Appliance configurations.

Sun Datacenter InfiniBand Switch 36
 Fixed 1u Switch
– Thirty six (36) QSFP InfiniBand Ports
– Data throughput: 2.3 Tb/sec (bidirectional)
– Port-to-port latency: 100 ns QDR
– Data virtual lanes: Eight
– Management virtual lanes: One
– MTU: 4,096 bytes
 Three in the SC
– One is the Spine switch
– Two are Leaf switches

InfiniBand Network on Half Rack

InfiniBand Network on Full Rack

Cisco Catalyst 4948E-F Switch
 Fixed 1u Switch
- 48 Ethernet Ports, each port wire-speed 10/100/1000 BASE-T
- 1+1 hot-swappable power supplies
- Back-to-front cooling
• Switch Capacity = 176 Gb/s
• Throughput = 131M Packets/s for IPv4
• IPv6 now hardware switched
• Used just for management
• Customers will be allowed to use a different ethernet switch

Management and Client Network

Card Locations on the SPARC T5-8 Server, Full Rack
 (1) Dual-port 10 GbE network
interface cards, for
connection to the 10 GbE
client access network
 (2) Dual-port InfiniBand host
channel adapters, for
connection to the InfiniBand
network

Card Locations on the SPARC T5-8 Server, Half Rack
 (1) Dual-port 10 GbE network
interface cards, for
connection to the 10 GbE
client access network
 (2) Dual-port InfiniBand host
channel adapters, for
connection to the InfiniBand
network

Exadata Storage Expansion Rack
 The optional Exadata Storage Expansion Rack
contains:
– Up to 18 Exadata Storage Servers
 4, 9, or 18 (Quarter-, Half-, or Full-Rack)
configuration
 High Capacity or High Performance storage
– Sun Data Center InfiniBand Switch 36 with 36
InfiniBand ports
 2 IB switches in the Quarter rack
 3 IB switches in Half and Full rack config

PDU Choices

Low Voltage Single Phase PDU
(Americas / Japan/ Taiwan)
 2 (Two) 110.4 Amp 22kVAPDUs
 Total of 6 Power Plugs (3 x 36.8 Amp Plugs per PDU)
 200 – 240VAC
 Data Center Receptacle Type - Hubbell CS8265C

Low Voltage Three Phase PDU
(Americas / Japan/ Taiwan)
 2 (Two) 120Amp 24kVAPDUs
 Total of 4 (Four) Power Plugs (2 x 60Amp Plug per PDU)
 200 – 208VAC 3ph
 Data Center Receptacle Type - IEC309-3P4W-IP67 60A 250VAC 3ph
(Hubbell equivalent is HBL460R9W)

High Voltage Single Phase PDU
EMEA /APAC (excluding Japan/ Taiwan)
 2 (Two) 96 Amp 22kVAPDUs
 Total of 6 (Six) Power Plugs (3 x 32 Amp Plugs per PDU)
 220 – 240VAC
 Data Center Receptacle Type - IEC309-2P3W-IP44 32A 250VAC

High Voltage Three Phase PDU
EMEA /APAC (excluding Japan/Taiwan)
 2 (Two) 109Amp 24kVAPDUs
 Total of 4 (Four) Power Plugs (2 x 25Amp Plug per PDU)
 220/380 – 240/415VAC
 Data Center Receptacle Type - IEC309-4P5W-IP44 32A 400VAC 3ph

Oracle SuperCluster
Software Stack
Applications
•Oracle, ISV and customer applications qualified on Solaris 10 or Solaris 11
Middleware
•Exalogic Elastic Cloud Software
Database
•Exadata Storage Server Software (ver 11.2.3.1)
•11gR2 Database Server Nodes (ver 11.2.0.3 DB + bundle patches)
Virtualization
•Zones (including Branded Zones) and LDoms
Operating System
•Oracle Solaris 11 for Oracle Database 11gR2
•Oracle Solaris 11 or Solaris 10 inApplication Domains for applications
Clustering
•Solaris Cluster 3.3 u1 for Solaris 10 and 4.0 for Solaris 11
Management
•Ops Center 12c (included) and Enterprise Manager Grid Control (optional)

Oracle SuperCluster – 1.0 to 1.1
• SSC 1.0 supported 9 “Pair-Wise” Configurations “A” - “I”
• SSC 1.1 introduced the “4 x 3” Domain Model:
• 4 = Maximum number of Domains on a single T5-4 Node
• 3 = Number of Domain Types (s10 App, s11 App, Database Domains)
• SSC 1.1 replaced 'solaris-desktop' install group w/ 'solaris-large-server '
• SSC 1.0 used spreadsheets to gather customer-specific config data
• SSC 1.1 used Python/Django GUI to gather customer-specific config data
• SSC-CU 1.0 uses Solaris ZFS Bootable Images to install LDom guests
• SSC-CU 1.1 uses Jumpstart & AutoInstaller to install LDom guests
• SSC-CU 1.0 took 2 days to run on-site
• SSC-CU 1.1 took less than 1 day
SSC Configuration & Installation Utility (SSC-CU)

Oracle SuperCluster – 1.1 to 1.5
• Major software overhaul too:
• Re-wrote large parts of SSC-CU to handle:
• T5-4 to T5-8 hardware changes
• Support a “State-File” like approach
• Re-wrote Config-O-Matic tool:
• To reflect hardware changes
• Deliver new configuration/state file format
• Modular approach to installation so code can be re-used for other tools
• iSCSI luns from ZFS SA used for guest Domains w/o local disk

Oracle Solaris 11. The First Cloud OS.
Next Generation of UNIX
 #1 UNIX
– Industry leading availability, security and
performance
 Built for Cloud Infrastructures
– Breakthrough architecture to deploy and secure and
manage enterprise clouds
 Engineered for Oracle
– Optimized for Oracle software and hardware

Enterprise Performance for Cloud Applications
Built for Next-Decade Hardware
 Dynamic threads
 NUMA I/O
 Crypto acceleration
 Latency-aware kernel memory
allocator
 Fully parallel network
processing
 Topology-aware scheduler
 Optimized shared memory
 Parallel network stack
 DTrace
 Adaptable thread and
memory placement
 NUMA-aware kernel memory fan-
out
 ZFS 128 bit block addresses
10x
CPU
10x
Memory
10x
Networking
10x
Data

Oracle Solaris 11
Built for Enterprise Clouds
Seamless Scaling with Hardware: 12 NEW World Records
Prioritized Threading + Next Gen I/O Terabyte of Memory + 10,000s of threads
SimplifiedAdministration: 3x faster provisioning, 4x faster updates
Cloud-designed installation + Fool-proof updates + Auto service case creation
Designed-in Virtualization: 4x lower latency, Zero overhead
Server, storage, network virtualization + Solaris 10 Zones + Zone clusters
Scalable Data Management: 128-bit FS, 5x storage savings
Integrated deduplication, compression + Infinite snapshots and clones + No cost replication
Advanced Protection: 3x faster encryption than Intel
HardwareAccelerated Encryption + Always-on auditing + Secure by default
2.4x Faster Database Performance
7x Better Middleware Price/Performance with Java

21st Century Cloud Infrastructure
For All Mission Critical Applications
Oracle VM
SPARC
Oracle Solaris 11 Oracle Solaris 10
Solaris 11
Zone
Solaris 11
Zone
Solaris 10
Zone
Solaris
Legacy Zone
Solaris
Legacy Zone
Solaris 10
Zone

SPARC Virtualization
Levels of Isolation
Server
Hypervisor
Solaris OSSolaris OS
Process
Process
Process
HardwareHardware
Dynamic Domains
OVM for SPARC
Solaris Zones
SuperCluster

• Oracle SuperCluster is configured according to customer requirements
– Database Domain (11gR2 today, 12c later)
– Application Domains
– Solaris 11 or Solaris 10
Configuring Oracle SuperCluster

• Oracle SuperCluster is configured according to your requirements
– Database Domain (Oracle Database 11gR2 today, 12c later)
– Application Domains (Oracle Solaris 11 or Oracle Solaris 10)

Configuration Flexibility
• Abroad set of configurations
– Complete configuration flexibility
– Up to 8 LDoms based on the supported domain types
– Simply choose the configuration you need
• Multiple domains of any type supported in a single node
• Zones supported in all domains, including Database Domains
– Multiple RAC and/or Single-Instance databases supported per Database Domain
• Fine-grained CPU/Memory allocation now supported
– 4 cores of CPU, 64GB memory (half rack) or 128GB memory (full rack)
• Zone root file systems can now be stored on iSCSI LUNs
– Located on the included Sun ZFS Storage 7320Appliance

• Some Simple Rules of Thumb Re Support
1. Don't mess with the hardware config unless it's specifically supported
2. Seek confirmation before changing any of
– Platform Configuration
– Includes ZFSSAconfiguration, IB configuration, etc
– Platform Software Releases
– Includes OS, firmware, Exadata Storage Servers, Oracle DB
– Versions should only be changed via the Quarterly Maintenance
Update, or when instructed by support
• Some things may be supported but not recommended
Is It Supported?

Versions of Oracle Solaris
 Depends on the type of logical domain created:
- Database Domain – For Oracle Database 11gR2 instances connected to
the Exadata Storage Servers, the domain must run Solaris 11
- Application Domain – For Oracle, 3rd party, or custom applications. Can
boot either Solaris 11 or Solaris 10. The Oracle Exalogic Elastic Cloud
Software must run in a Solaris 11 App Domain.
 Solaris 10 Branded Zones are supported in an Application Domain running
Solaris 11
 Solaris 8 and 9 Branded Containers are supported in an Application Domain
running Solaris 10

New Features: Domain Types
Overview
1Full rack (4 for half rack) 2Full rack (3 for half rack)
Domain Type Domain Footprint
on Oracle
SuperCluster
Access to
Exadata
Storage
Servers
Access to Shared
Storage on Included
ZFSSA
SAN access via
Optional Fiber
Channel HBAs
Suitable
to host
Exalogic
Software
Zones
Support
Database Domain Zero to eight1
domains per
SPARC T5-8 node
Yes Can be custom
configured post install
Supported No Yes
Application
Domain
booting Oracle
Solaris 11
Zero to eight1
domains per
SPARC T5-8 node
No Configured for optional
Exalogic Elastic Cloud
Software, general
application binaries,
and logfiles
Supported Yes Yes
Application
Domain
booting Oracle
Solaris 10
Zero to seven2
domains per
SPARC T5-8 node
No Configured for general
application binaries and
logfiles
Supported No Yes

Database Domain
• Runs Oracle Database 11gR21 instances
• Supports both RAC and single-instance databases
• Configuration Specifics
• Up to eight Database Domains supported per SPARC T5-8 node
• For full rack; up to 4 domains for half-rack
• Entire SPARC T5-8 node can be dedicated to Database Domain(s)
• Only the Database Domain has access to Oracle Exadata Storage Servers
• Dynamic Virtualization
• Oracle Database 11gR2 instances (RAC or Single Instance) supported in Oracle
Solaris Zones in the Database Domain
1Licenses are required for Oracle Database 11gR2

Oracle SuperCluster with No Database Domains
• Database Domains are always required, apart from the following exceptions
1. Oracle Exadata Storage Servers included with a Oracle SuperCluster may be
managed from an existing Oracle Exadata
• Database instances can be consolidated on the Oracle Exadata and
therefore need not be present on the Oracle SuperCluster rack
• The Oracle SuperCluster and the Oracle Exadata must be co-located and
connected on the same InfiniBand fabric
2. Multi-rack Oracle SuperCluster configuration
• Database instances can be consolidated on a subset of SuperCluster racks
• Note: Oracle Exadata Storage Servers included in the Oracle SuperCluster
configuration must always physically remain in their original racks, even if
managed from another rack

Database Domain Storage
• Oracle Exadata Storage Servers1
• Dedicated high performance storage for Oracle Database 11gR2 instances
• Optimal Oracle Database 11gR2 performance on SuperCluster depends upon
leveraging the unique benefits of Oracle Exadata Storage Servers
• Additional storage servers via Oracle Exadata Storage Expansion Rack
• Both High Performance and High Capacity options available
• Normal redundancy (double mirroring) standard for half-rack configs
• High redundancy (triple mirroring) requires min of 5 storage servers
1Licenses are required for Exadata Storage Server disks

Database Domain Storage
• Sun ZFS Storage 7320 Appliance
• Accessible from Database Domain
• Can be used for RMAN backups or flatfile staging
• No shares configured by default
• Optional Fiber Channel HBAs
• Supported in the Database Domain

Oracle Solaris 11 Application Domain
• Runs any application supported on SPARC/Oracle Solaris 11
• Well suited to host Exalogic Software1
• Domains with single T5 socket/processor recommended
for best performance with Oracle Exalogic Elastic Cloud software
• Enterprise applications and middleware
• Oracle Database 11gR2
• Earlier Oracle Database releases are supported only on Oracle Solaris 10
• Non-Oracle databases supported on Oracle Solaris 11
• Domains of this type are optional
• ASPARC T5-8 node can be dedicated to an Application Domain running Oracle Solaris 11
1Licenses are required when installing optional Exalogic Elastic Cloud Software

• Oracle Solaris Zones offer a further layer of flexible virtualization
• Oracle Solaris 10 Branded Zones can be used to host Oracle Solaris 10
applications
• Oracle Solaris Cluster
• Oracle Solaris Cluster supports clusters of Oracle Solaris 11 nodes on Oracle
SuperCluster
• Note: Oracle Solaris 10 and Oracle Solaris 11 nodes cannot be mixed in the same
cluster

Oracle Solaris 11 Application Domain Storage
• Application binary and logfile shares configured by default
• Supported in Oracle Solaris 11 Application Domains

• Runs full range of applications supported on SPARC/Solaris 10
• Enterprise applications and middleware
• Oracle Database releases, including 11gR2, 11gR1, 10g, 9i,…
• Non-Oracle databases (e.g. Sybase, DB2, MySQL,…)
• Domains of this type are optional
• ASPARC T5-8 node can be a singleApplication Domain running Oracle Solaris 10
• Apart from a thin Oracle Solaris 11 Control Domain (the Control Domain must
run Oracle Solaris 11)
1Licenses are required when installing optional Exalogic Elastic Cloud Software

• Oracle Solaris Containers offer a further layer of flexible virtualization
• Oracle Solaris 9 applications supported in Oracle Solaris 9 Branded Zones
• Oracle Solaris 8 applications supported in Oracle Solaris 8 Branded Zones
• Oracle Solaris Cluster
• Oracle Solaris Cluster supports clusters of Oracle Solaris 10 nodes on Oracle
SuperCluster
• Note: Oracle Solaris 10 and Oracle Solaris 11 nodes cannot be mixed in the same
cluster

Oracle Solaris 10 Application Domain Storage
• Application binary and logfile shares configured by default
• Supported in Oracle Solaris 10 Application Domains

Configuration Options
• Broad Range of Configurations Supported
• Tested at Oracle
• Delivered via standard ACS Oracle SuperCluster install
• Implemented by Oracle SuperCluster Configuration Utility (SSC-CU)
• Post-Install Configuration Changes
• Major changes can be supported by ACS on T&M basis
• Applies to domain reconfiguration
• CPU and Memory allocation changes can be carried out directly by customers

Configuration Options
• Customer chooses the configuration
• Identify the configuration for each T5-8 node that best matches intended deployment
• Selection is done as part of pre-install process
• Database Domains must be included in every rack (apart from previously stated
exceptions)
• Selected configuration(s) are implemented during install
• Carried out by Oracle ACS using Oracle SuperCluster Configuration Utility
Half and Full Rack
Select up to two configurations*
OR
Select one configuration,which will be applied to both SPARC T5-8 nodes
*ConfiguringSPARC T5-8 nodes in pairs makes sense in most cases

CPU and Memory Layout
• CPU and Memory
• At install time, allocation granularity is 1 socket per domain
• 16 CPU cores and 128GB memory
• Delivers optimal performance
• Finer-grained CPU and memory allocation
• CPU and memory can be reassigned after the install
• Optimal flexibility
• Carried out by a standalone tool, setcoremem
• CPU and Memory can be assigned independently
• Granularity: 4 CPU cores, 64GB memory (half rack) / 128GB (full rack)
• Node reboot will be required

Additional Database License Requirements
• Software licenses are required for the database in Database Domains
• Based on standard SPARC T5 per-core multiplier for cores allocated to
Database Domains
• Licenses are not required before Oracle Database 11gR2 is used
• Oracle Exadata Storage Servers
• Exadata Storage Server disks must be licensed
• Aminimum number of disks must be licensed
• 24 disks (2 Oracle Exadata Storage Servers) for single-instance databases
• 36 disks (3 Oracle Exadata Storage Servers) for RAC databases
• Licenses are not required before Oracle Exadata Storage Servers are used

Additional Application License Requirements
• Oracle Exalogic Elastic Cloud Software
• Software licenses are required if the optional Oracle Exalogic Elastic Cloud
Software (EECS) is deployed
• Based on standard SPARC T5 per-core multiplier for cores allocated to domains
running EECS
• Oracle Middleware and Applications
• Subject to Oracle’s standard core-based licensing, applying the SPARC T5 per-
core multiplier (0.5)
• Capped Containers/Zones are recognized as partitions for licensing purposes

Domain to PCIe Slot Mapping: Half Rack

Domain to PCIe Slot Mapping: Full Rack

Power and Cooling: Half Rack
SSC T4-4
SC T5-8

Power and Cooling: Full Rack
SSC T4-4 SC T5-8

Licensing By Cores
 Not all cores must be assigned
 Unused cores are considered parked cores and are not counted for
licensing
 Parked cores are unusable and essentially wasted
– assign cores to a different domain is better approach

Database Options for Oracle SuperCluster
 Oracle Database 11gR2
- Runs in dedicated Database Domains, storing data on the Exadata Storage
Servers
- Can be run in Application Domains running either Solaris 11 or Solaris 10
 Earlier Oracle Database releases (11gR1 and earlier)
- Can be run in Application Domains running Solaris 10
- Can be run in a Solaris 10 Branded Zone in either the Application or
Database Domains
 Non-Oracle databases
- Can be run in Application Domains running either Solaris 11 or Solaris 10,
in both global amd local zones

Oracle Exalogic Elastic Cloud
Extreme Performance for Middleware & Applications
BreakthroughArchitecture
–10x latency improvement for database access
–15x faster session replication
–10x faster response time
Accelerated Java
–2 Million JMS messages/sec
–60% more Java ops/sec

Exalogic on Oracle SuperCluster
 The following features of the Exalogic Elastic Cloud Software are
available for Oracle SuperCluster:
- WebLogic JDBC and Data Source Optimizations
- WebLogic Cluster State Replication Optimizations
- WebLogic Socket Direct Protocol (on the Oracle SuperCluster InfiniBand
stack only)
- Coherence message bus API support (on the Oracle SuperCluster
InfiniBand stack only)

In Conclusion
• Flexible Configuration
• Features optimal performance
• Integrates Oracle Database 11gR2 with Oracle Exadata Storage Servers
• Integrates optional Oracle Exalogic Elastic Cloud Software
• Caters for key customer deployment environments
• Tested at Oracle
• Accelerated Deployment
• Relieves sysadmins of the routine work
• Vastly reduces provisioning time and effort
• Oracle SuperCluster
• Transforms complexity into simplicity

Differences Between Exalogic and SuperCluster
 Oracle SuperCluster incorporates different technologies:
- Connectivity to Data Center
 Ethernet over IB on InfiniBand Gateway Switches on Exalogic
 Multiple 10GbE NICs on each SPARC T5-8 on Oracle SuperCluster
(along with InfiniBand HCAs connected to InfiniBand Switches)

– Software licenses are required as soon as the database is used
– Based on standard SPARC T5 per-core multiplier for cores allocated to the Domain(s)
• Oracle Exadata Storage Servers
– Exadata Storage Server disks must be licensed
– Aminimum of 24 disks (2 Oracle Exadata Storage Servers) for single-instance databases
– Aminimum of 36 disks (3 Oracle Exadata Storage Servers) for RAC databases
– Licenses are not required before Oracle Exadata Storage Servers are used
• Oracle Exalogic Elastic Cloud Software
– Software licenses required if optional Oracle Exalogic Elastic Cloud Software (EECS) is deployed
– Based on standard SPARC T5 per-core multiplier for cores allocated to domains running EECS
License Requirements

Getting Data to the Oracle SuperCluster T5-8
 Optional FC
– Up to 8 optional FC cards can be added to each T5-8 node in the half rack
configuration
 The full rack configuration requires that a 10GbE card be replaced with the FC
card
– Fibre Channel PCIe for connectivity to legacy SAN environments
– Not included as standard component; requires a separate line item
– Supported for connectivity from existing SAN storage
– The following cards are supported:
 SG-XPCIE2FC-EM8-N - Sun StorageTek 8Gb FC HBA in low profile form factor, Emulex
 SG-XPCIE2FC-QF8-N - Sun StorageTek 8Gb FC HBA in low profile form factor, Qlogic

Use of Fibre Channel on Oracle SuperCluster
 FC/SAN configurations supported with Oracle SuperCluster T5-8 are the same as those
supported with regular SPARC T5 servers
 FC HBAs may be used for any purpose supported with FC HBAs on regular SPARC T5
servers
– FC HBAs may be placed in Application Domains and/or Database Domains

Oracle Enterprise Manager
Ops Center 12c
 Discover Oracle SuperCluster
as a single system
 Hardware event interface view
of Oracle SuperCluster as a
single system
 Monitoring and active
management tasks separated
by role
Total Systems Management

Oracle SuperCluster Ops Center
 Unified management of
Servers, Storage, and
Network Fabric
 Simplified management
of virtual infrastructure
for easy application
consolidation
 Instant Network and
Storage provisioning
 Automated update of all
firmware and software
components
 Direct connection to
Oracle knowledge-base
speeds problem
resolution

Oracle SuperCluster Ops Center

SPARC Security
Securing the Stack Layers
 Secure database queries 43% faster; Low overhead for encryption
Results  WebLogic SSL & WS-Security 2x – 3x faster
 ZFS Filesystem Crypto 3x faster than x86
 Java crypto 2x – 3x faster than x86
 OpenSSL4.3x faster for single-thread security versus Power7

Oracle SuperCluster Toolkit
Installation and Configuration Monitoring and Assistance
 Oracle SuperCluster Configuration Utility
 SC DCLI (Distributed Command Line Interface)
 SC Topology Verifier
 SC Check
Saves thousands of lifetime hours of
error-prone testing, tuning,
configuration and optimization

Oracle SuperCluster Deployment
Hardware, Software
and Install Service
• System Installation and
Configuration Services
• Starter Pack, Install and
Configuration, etc.
• Automated Service Requests
Simple and Efficient
“Phone Home”
Typical
4X
Deployment Speed Up
All Applications
• Deploy Oracle, 3rd Party, Java,
and Custom Applications
• My Oracle Support Integration
• One-Patch Packaging

Seamless Scalability
Start Small. No Limits.
Existing Data
Full Rack
Half Rack
Multi-Rack

Upgrading From Half To Full Rack
Meet growing database/application requirements
The upgrade hardware includes:
– Components for T5-8 compute servers
 4 x SPARC T5 processor modules, for a total of 128 cores
 128 x 16GB total memory
 8 x InfiniBand cards
 8 x 10GbE cards
– 4 x Exadata Storage Servers X4-2 with 12 x 1.2TB 10,000 RPM High
Performance disks or 12 x 4 TB 7,200 RPM High Capacity disks
– InfiniBand and Ethernet cables to connect all the components

Highly Scalable Engineered Systems
Easily Mix and Match To Meet Your Needs
Oracle SuperCluster Any combination Oracle SuperCluster, Exadata Storage Expansion
Rack, Exadata, Exalogic, and Sun ZFS Storage Appliance

Agenda

Primary Use Cases for Oracle SuperCluster
 Most efficient engineered system to consolidate SPARC servers
 Superfast engineered system to run multiple enterprise applications
 Lowest cost platform for Oracle customers to consolidate HP and IBM
systems

Oracle SuperCluster
Deployment Flexibility
Oracle SuperCluster allows deployment of
 Multiple tiers of an application
 Multiple applications
– E.g. Payroll, Supply Chain, SAP Customer Service
 Multiple Databases
Applications can run on a mix of Solaris releases
 Oracle Solaris 10
 Oracle Solaris 11
 Older releases supported with Branded Zones

T5-8 Node 2
Solaris 11
Solaris 11
DB Domain
App Domain
T5-8 Node 1
Solaris 11
Solaris 11
DB Domain
App Domain
Exa
data
• Valid configuration, but requires custom installation.
Exa Exa
data data
Exa
data
Oracle SuperCluster - How Is It Used?
InfiniBand
•Built-in virtualization
•Runs wide-range of
workloads
•Runs Solaris 10 and 11
•Runs multiple
databases
•Runs multiple
applications and tiers
ZFSSA
N
e
twork
Exa
data
Exa
data
Exa
data
Exa
data

Exa Exa Exa Exa Exa Exa Exa Exa
data data data data data data data data
Oracle SuperCluster - How Is It Used?
T5-8 Node 1
Solaris 11
T5-8 Node 2
Solaris 10
Solaris 11
InfiniBand Network
DB Domain
App Domain
App Domain
ZFSSA
•Built-in virtualization
•Runs wide-range of
workloads
•Runs Solaris 10 and 11
•Runs multiple
databases
•Consolidates multiple
applications and tiers

Oracle SuperCluster - Internal Clusters
Full Rack with HA Example
DB/Exadata Storage Servers + Solaris 10 Apps use case
 Entire nodes dedicated to a single stack in this example
Node 1 Node 2
Oracle Solaris10
App Domain
Oracle Solaris10
Siebel
ISV app
Oracle Solaris 11
DB Domain
Oracle Solaris10
App Domain
Oracle Solaris10
Siebel
ISV app
Oracle Solaris 11
DB Domain

Sizing Oracle SuperCluster Systems
 Migrating older systems to Oracle
SuperCluster
– T5-8 typically is able to replace
multiple older servers
 Sizing only based on performance
comparisons
 Other factors to consider
– Serviceability (i.e. Hot-plug capability)
– Reconfiguration impact on domains
Based on performance alone, a full
rack can replace:
22 M5000 (8 * 2.66GHz)
10 M8000 (16 * 3.0GHz)
11 Sun Fire E25K (72 * 1.95GHz)
247 E3000-E6500 (30 * 464MHz)
85 F3800-F6800 (24 * 1.2GHz)
24 Sun Fire E6900 (24 * 1.95GHz)
66 Sun Fire V890 (8 * 2.1GHz)
61 T5x20 (6 core) (1 * 1.2GHz)
19 T5440 (4 * 1.6GHz)

Oracle Benchmark Focuses -- Customer Value
Based on deep understanding of Enterprise Applications & Database
 Sizing & performance should focus on realistic benchmarks
• Oracle focuses on realistic workloads and system comparisons
• Some benchmarks do not map to business needs
EBS, PeopleSoft, JD Edwards, Siebel
SPECjEnterprise
TPC-H Max Customer Value
TPC-C
SAP-SD benchmark
More ABAP interpreter than typ cust
SPECjvm
SPECint_rate
SPECjbb
Rapid decrease in
I/O, network & compute
Rapid decrease in
Fit for real workloads

Oracle SuperCluster
In Conclusion
Oracle SuperCluster
 Is an ideal platform to support full-stack applications
 Offers all the benefits of an Oracle engineered system
– Integrated compute and storage with an InfiniBand backplane
 Offers a logical upgrade path for existing SPARC customers
 Delivers a compelling & highly competitive alternative to IBM & HP
– Superior performance at a significantly lower price
– Sysadmin savings on initial installation and ongoing support
– Reduced Total Cost of Ownership

One Engineering Team
Best Performance. Best Value.
Oracle
Engineered
Systems
Compute &
Storage Building
Blocks

Presentation oracle super cluster t5-8 technical deep dive

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Presentation oracle super cluster t5-8 technical deep dive

Similar to Presentation oracle super cluster t5-8 technical deep dive (20)

More from solarisyougood

More from solarisyougood (20)

Recently uploaded

Recently uploaded (20)

Presentation oracle super cluster t5-8 technical deep dive