Oracle SPARC T4-4 Technical Overview

<Insert Picture Here>
SPARC T4-4 System Technical Overview
Download this slide
http://ouo.io/ocZIa

The following is intended to outline our general
product direction. It is intended for information
purposes only, and may not be incorporated into any
contract. It is not a commitment to deliver any
material, code, or functionality, and should not be
relied upon in making purchasing decisions. The
development, release, and timing of any features or
functionality described for Oracle’s products remains
at the sole discretion of Oracle.

Oracle SPARC T4-4 System
• Compute
• 2x or 4x SPARC T4 8-core CPU (128/256threads)
• 64x DDR3 DIMMs (1.024 TB max, with 16GB DIMMs)
• I/O and Storage
• 16x Express Module PCIe Gen2 x8 slots
• 8x 10GbE XAUI ports (2 x QSFP connectors)
• 8x 2.5” hot-swap SAS-2 disks
• 300GB, 600GB HDD; 100GB, 300GB SSD
• Availability and Management
• RAID 0/1/1E
• Hot-swap fans, disks & PSUs
• ILOM Service Processor
• Electronic Prognostics 1.2
4-Socket, 5RU Enterprise-Class Data center Server

SPARC T4-4
System Highlights
• Successor to SPARC T3-4 system
• Uses Express Modules for PCIe
• 5U enclosure with PCIe Generation 2.0 and SAS2
Disk Drives
• Same Service Processor module as T3-based
systems
• Solaris 10 8/11 (Oracle Solaris 11 post-Announce)
• Oracle VM Server for SPARC 2.1 (formerly known as
Logical Domains)
• Wire Speed Cryptography: 12 popular algorithms;
SPU instructions now implemented within each core’s
pipeline

SPARC T4-4
Positioning
• Workload Segments and Deployment Scenarios
• Database and data warehousing/BI
• Back Office (ERP, CRM, Supply Chain, Accounting,
Lifecycle Mgmt, OLTP)
• Internet Infrastructure (web, mail, app/j2ee, db, proxy,
directory, media)
• Large, single-app implementations and/or multi-app
consolidation (LDOMs, Containers)
• Popular Applications
• Oracle 11g, Siebel 7.7.1 CRM and Siebel Analytics 7.8.4,
Baan ERP, mySAP ERP (R/3), Websphere
• BEA Weblogic Server 9.0, Apache, Symantec Brightmail &
ScanEngine, Sybase IQ & ASE 12.6
• NetBackup Media Servers (SAN-> Tape), Sun JES (web,
email, dir), PeopleSoft 8.4, PeopleTools 8.46

SPARC T3-4 Comparison SPARC to T4-4
Feature SPARCT3-4 SPARCT4-4
FormFactor 5RU,28”deep 5RU,28”deep
CPU
2xor4xSPARCT3
1.65GHz (256or512threads)
2xor4xSPARCT4
3.0GHz(128or256threads)
Memory
DDR3,512GBMAX
64xSlots
DDR3,1.024TBMAX
64xslots
Network
4x1GbE+8x10GbE(XAUI)
Requires2SeparateQSFPConnectors
4x1GbE+8x10GbE(XAUI)
Requires2SeparateQSFPConnectors
InternalStorage
Upto8x2.5”SAS
(SSDsupportTBD)
Upto8x2.5”SAS,canuseupto
4xSATASSDs,hot-swap
RemovableMedia 1x DVD-RW(viarKVMS;notlocal) 1x DVD-RW(viarKVMS;notlocal)
Serial 1xRS-232, 4xUSB,1xVGA 1xRS-232, 4xUSB,1xVGA
PCIExpressslots 16xx8slots(ExpressModuleslots,Gen2) 16xx8slots(ExpressModuleslots,Gen2)
PowerSupply
4x2060WattAC,N+N
Redundant/Hot-Swap
4x2060WattAC,N+N
Redundant/Hot-Swap
Fans 5xRedundantHot-Swap 5xRedundantHot-Swap

SPARC T4-4 System
Key Features
• SPARC T4 CPU
• SPARC T4 based on SPARC T3: new core but same SoC
• 8 newly-designed cores, dual 10 Gbps Ethernet, dual PCIe Gen2
• 2x or 4x T4 processors per system, 8 cores x 8 threads → 128 or
256 threads per system
• Core frequency of 3.0 GHz
• T4 chip is socketed, but not field-replaceable
• 64 sockets on 2 Processor Modules Boards → up to 1.024 TB (w/
16GB DIMMs)
• 4 GB, 8GB, and 16GB DIMM sizes (32GB DIMMs post-Announce)
• Memory – DDR3 DIMMs, Registered, ECC, operating at 1066
MT/s
• Next Generation Buffers-on-Board (BoB) between CPU and
DIMMs

SPARC T4-4 System
Key Features
• Software
• Solaris 10 8/11 (Oracle Solaris 11 post-Announce)
• Oracle VM Server for SPARC 2.1
• Service Processor (SP)
• SP module, shared across T4-based platforms
• AST2200 CPU, with local Flash and memory
• ILOM 3.0
• Electronic Prognostics 1.2
• Parallel Boot
• Host and SP boot in parallel → faster boot time
• Support for “degraded” modes:
• with failed Service Processor
• with failed T4 Processor(s)

SPARC T4 Systems
I/O Subsystem
• PCIe Gen 2.0 with 2 x8 lane physical ports per SPARC
T4
• Full bandwidth per port with Generation 2 speeds
• PCIe 2.0 to PCIe 1.0 autonegotiation
• SDIO supported now, Single Root IOV planned for Oracle VM
Server for SPARC (post-Announce)
• 4 GBps each direction, total of 16 GBps bandwidth per
SPARC T4
• Lane width autonegotiation
• 256 byte maximum packet size
• Uses latest PCIe switch chips from IDT

SPARC T4 Systems
I/O Subsystem
• 4th Generation Virtual to Physical Address Translation
• 52-bit virtual address, 48-bit physical address
• 2GB page support will be integrated into Solaris 10 8/11
• TLBs are fully associative; 2GB entries shared with
other page size entries
• ISM / DISM will get 2GB page size by default if request
is aligned, large enough and the pages are available

SPARC T4-4 System
Key Features
• I/O – Networking
• 1GbE
• 4 on-board ports, 10/100/1000 Mbps
• Two Intel 82576 dual-port MAC+PHY
• On-board ports also support NC-SI to Service Processor
• 10GbE
• Up to 8x 10GbE XAUI links from SPARC T4
• XAUI ports operate independently of on-board ports
• XAUI requires QSFP connections, 10GbE Board is
standard

SPARC T4-4 Systems
Key Features
• I/O – Storage
• HDDs/SSDs
• HDD: 300GB & 600GB at Announce
• SSD: 100GB & 300GB at Announce
• Dual LSI SAS2008 8-port SAS2/SATA2 controllers, RAID
0/1/1E
• No DVD on T4-4; use rKVMS

SPARC T4-4 Systems
Key Features
• I/O – Expansion
• 2 PCIe Gen2 switches, each directly accessible from two
SPARC T4s, one from each of the two Processor Modules
• 16 Express Module PCIe expansion slots
• All slots accept x8 PCIe Module cards, Gen1 or Gen2
• USB 2.0
• 1 Internal storage device port (unsupported, but useable),
2 front ports, 2 rear ports
• SIS buttons/LEDs, Service Processor Serial/Ethernet ports,
VGA port

SPARC T4-4 Front Panel
DISK 0
DISK 1
DISK 2
DISK 3
DISK 4
DISK 5 DISK 7
DISK 6
Dual USB 2.0
Ports
HD-15 VGA
Port
SP Serial
Port
Processor
Module 1
PSU Status
LEDs
PSU 0 PSU 1 PSU 2 PSU 3
RFID/Serial
Number
Main Module &
SP Status LEDs
Power
Button
Status
LED
Fault
LED
Locator
LED/Button
Processor Module 0
Status LEDs
Over Temp
Indicator
Rear Fan/EM
Indicator
Processor
Module 0
Processor Module 1
Status LEDs
Main Module
(Entire Board)

SPARC T4-4 Rear Panel

SPARC T4-4 Block Diagram

SPARC T4-4 System Chassis
• 5U system chassis
• 5 fan modules each front-to-back fans
• Main Module board, contains HDDs/SSDs, SP
• Processor Module board, contains SPARC T4 CPUs (cooling
via copper heat sinks) and memory
• Uses same Power Supplies as T3-4 server

© 2011 Oracle Corporation
19
SPARC T4-4 Rear Chassis Assembly
Midplane, Express
Backplane, PDB,
AC and DC Busbars
(part of Rear Chassis
Assembly)
Rear Chassis Sheet metal
sleeve, Fan and Card Guides
System Chassis

20
T4-4 Rear Chassis Assembly
Midplane, Express
Backplane, PDB,
AC and DC Busbars
(part of Rear Chassis
Assembly)
Rear Chassis Sheet metal
sleeve, Fan and Card Guides
Rear Chassis
Sub-Assembly
Entire assembly serviced
from rear
System Chassis

SPARC T4-4 System Power
• AC Power Supply
• “2U” (78 mm) A239
• 2060W @ 240 VAC
• Gold+ efficiency
• N+N Redundant, Hot-swap
• 4 Power Supplies
• DC/DC Voltage Converters
• D219 for DDR3 memory – 1.35 V Vdd, 0.75 V Vtt
• D220 for CPU – 1.05 V (nom.) Vdd, Vnw, Vsb
• Shared with all T4 platforms

22
SPARC T4-4 PSU and System Power
PSU (A239)
• A minimum of two A239 PSUs are required for
each system (2060 W each)
• They may be placed in any Power Supply Bay
(front of system)
• T4-4 will only ship to customers as 200-240V
(high-line) system
• AC cords (4) plug into rear.

23
SPARC T4-4 System Service processor
• Common service processor card for all systems
• Aspeed AST2200 processor
• 266MHz ARM processor
• DDR2
• PCIe video device PCIe 1.0 x1
• USB
• Provides rKVMS
• 10/100Mbps network interface and serial console
• ILOM 3.0

SPARC T4-4 Service Processor

Service processor is separate FRU

Requires removal of Main Module from
chassis, then removal of mezzanine-like
board from Main Module

SPARC T4-4 Main Module Internal View
Service
Processor
C
C o
h n
a n
s e
s c
i t
s o
r
s
System
Configuration
Card (SCC)
EEPROM
8x 2.5 “
HDDs
Or
SSDs
REM0 and
REM1 Raid 0/1
Cards
PCIe Gen2
Switches

SPARC T4-4 Processor Module w/DIMMs &
Fillers
DDR3
DDR3
DDR3
DDR3
DDR3
DDR3
DDR3
DDR3
SPARC
T4MODULE
SPARC
T4MODULE
C
C o
h n
a n
s e
s c
i t
s o
r
s
Buffers-on-
Board (BoB)
Chips

SPARC T4-4 Processor Module w/out
DIMMs
DDR3
DDR3
DDR3
DDR3
DDR3
DDR3
DDR3
DDR3
SPARC
T4MODULE
SPARC
T4MODULE
C
C o
h n
a n
s e
s c
i t
s o
r
s

T4-4 Basic Memory Layout
•Each SPARC T4 CPU
contains 2 memory
controllers
• MCU0 and MCU1
•Each controller has
interfaces to 2 Buffers on
Board (BoBs)
• 2 DDR3 channels
• 2 DIMMs per channel
• 16 DIMMs per CPU
•Memory bus speed limited
to 1066 MHz for all
platforms

Processor Module Physical Memory Layout

SPARC T4-4 Memory Configuration
Rules
• Minimum of 16 DIMMs per Processor Module; 4GB, 8GB
and 16GB DIMMs are supported
• 8 DIMM configurations are recommended to provide full
memory bandwidth and optimal performance
• 32GB DIMMs available post-Announce
• Minimum capacity: 64GB
• Maximum capacity: 1.024TB (2.048TB post-Announce)
• All DIMMs run at 1066 MHz, no clocking down

SPARC T4-4 10GbE XAUI
• RIO Board and QSFP
with connectors
• Direct interface to dual
10GbE ports on each
SPARC T4
• Standard with ATO order from
factory:
•SE5Y5RM1Z: Snap-in RIO Module
(native: 4 x 10/100/1000 MbE,
SP-Network, SP-Serial, 2 x USB 2.0),
8 x 10GbE XAUI ports via 2 QSFP
connectors

SPARC T4-4 Disk Drives
• Disk drives are 2.5” form factor
• 5U Chassis
• Chassis supports SAS2
• Up to 8 SAS drives
• 300GB @ 10K RPM
• 600GB @ 10K RPM
Ready to
Remove
Fault
Status
Disk LED's

SPARC T4-4 SSD Drives
• 100GB SSD available at Announce
• 300GB SSD available post-Announce

SPARC T4-4 Disk Controller
• T4-4 has Dual LSI SAS2008 8port SAS2/SATA2
controllers
• Support for RAID 0 (striping), RAID 1 (mirroring) using
'raidctl’ and RAID 1E (enhanced mirroring)

SPARC T4-4 Rack Mounting
• Only one rack mounting solution – standard with base
chassis
• Rail slide kit for rack mounting (factory installation)
• Order # (Base Chassis Part Number): SE5ASY11Z

SPARC T4-4 System RAS Overview
• Designed to minimize part count and operating
temperature to enhance reliability
• End-to-end data protection detecting and correcting
errors throughout server – ECC everywhere
• Processor and Memory protection
• CPU core and thread off-lining
• Memory with ECC, x4/x8 DRAM Extended ECC, page
retirement, and lane failover
• Major components redundant & hot-replaceable
• Fan, Power Supply, and internal disks
• RAID capability for internal disks

Linux KernelLinux Kernel
Guest MgrGuest Mgr
UBoot/DiagsUBoot/Diags
Service Processor
ILOM
Host ConfigHost Config
HypervisorHypervisor
- Environmentals
- Fault Management
- LED Control
- SP Diags
- DFRUID
- Plat HW Svc
- FMA ETM
- IPMI
- CLIs
- Logs
- SNMP
- FMA Support
- PowerOn/Off
- FERG
CPU (AST2200)CPU (AST2200)
CPUCPU MemoryMemory IOIO
Host
LDC Channels
- Host Config
- Machine Description
- Hypervisor
- OBP
- POST
Host Flash
OBPOBPOBPOBP
Solaris
S10U10
Solaris
S10U10
sun4vsun4v
Solaris 11
Express
Solaris 11
Express
sun4vsun4v
OBPOBP
POSTPOST
System
Domain
System
Domain
LDOMS
Manager
sun4vsun4v
Host Data Flash
- OBP NVRAM/POST/SC config vars
- ASRDB -LDOMS config
- Console Log - SER log – TOD data
Host Data Flash
- OBP NVRAM/POST/SC config vars
- ASRDB -LDOMS config
- Console Log - SER log – TOD data
Platform Hardware
FPGAFPGA
- Kernel
- FMA Components
- Platform Drivers
- Kernel
- FMA Components
- Platform Drivers
- S10U10
- FMA Components
- Platform Drivers
SPARC T4-4 Software/Firmware Block
Diagram

© 2011 Oracle Corporation Page 38
Oracle Solaris and SPARC Virtualization
Better Resource Utilization for a More Efficient Data center
Dynamic Domains Oracle VM Server
for SPARC
M-Series T-Series
App App
Oracle Solaris
Containers
Oracle Solaris
DW DB
Domain A
Domain B
OLTP DB
OLTP DB
App
App
Domain A
Domain B
Domain C
Web
Web DB App Web
Web
Web

Oracle VM Server for SPARC 2.1
Customers benefit from increased application service level
• New Hardware Support
• SPARC T4 servers with Oracle Solaris 10 8/11; Solaris 11 post-Release
• Secure live migration
• Dynamic resource management (DRM) between domains
• Integrated Dynamic Reconfiguration (DR) of cryptographic
units and virtual CPUs.
• Increased maximum number of virtual networks per domain
• Support for virtual device service validation
• Lower-overhead, higher scalability networking for Oracle
Solaris 11 initial release
• Enhanced Management Information Base (MIB)
• Physical to virtual (P2V) enhancements

Secure Live Migration
• Live migration available on
SPARC T-Series systems
• SPARC T4
• SPARC T3
• UltraSPARC T2 Plus
• UltraSPARC T2
• On-chip crypto accelerators
deliver secure, wire speed
encryption for live migration
• No additional hardware required
• Eliminates requirement for
dedicated network
• More secure, more flexible
VM
External Shared Storage
SPARC T-Series servers
Oracle VM Server Pool
VM Secure Live Migration (SSL)VM VM
Eliminates Application Downtime

Live Migration Requirements
• Source System
• Guest domain with virtual I/O devices only, running Solaris 10 9/10
• Requires Logical Domains Manager 2.1 & updated firmware (i.e.
7.4 or 8.1)
• Can not have multi-pathed disks (IPMP for networks is ok)
• Power Management in “performance” mode (the default)
• Target System
• Must have enough resources (cpu, mem)
• Must have appropriate VIO services (vds, vsw, vcc)
• Must be able to provide required VIO devices (vdisk, vnet)
• Must be cpu-compatible
• Same processor type (e.g. SPARC T4), same clock frequency

Perform Live Domain Migration
• Uses the same CLI and XML interfaces as in prior
releases
• Also from Oracle Enterprise Manager Ops Center
• CLI example
– ldm migrate [-f] [-n] [-p <password_file>] <source-ldom>
[<user>@]<target-host>[:<target-ldom>]
• -n : dry-run option
• -f : force
• -p : specify password file for non-interactive migration
• Cancel an On-Going Migration
– ldm cancel-operation migration <ldom>

Live Migration Best Practices
• There is no specific requirement on the number of CPUs
in the control domains. However our experience shows
that > 8 vCPUs is best. We recommend 16 or more
vCPUs in order to minimize the suspend time as well as
the overall migration time.
• Workloads that heavily modify memory pages will have
longer migration times
• Be sure to add the crypto units for best migration
performance.
• Review the documentation, especially the Admin Guide
for planning live migration

Resource Management Improvements
• Dynamic resource management (DRM) between domains
• Dynamic CPU movement is based on the priority property of each
domain's DRM policy.
• Ensures that domains running the most important workloads get
priority for CPU access over domains with less critical workloads
• Integrated Dynamic Reconfiguration (DR) of Cryptographic
units and virtual CPUs
• Automatically remove crypto unit when the final vCPU of a core is to
be removed.
• Simplify operations and ensure consistent performance.

Increased Maximum Number of Virtual
Networks Per Domain
• Introduces an option to dynamically disable the allocation
of inter-vnet Logical Domain Channels (LDC)
• The number of LDCs is limited by hardware.
• Inter-domain communication continues to work exactly as before,
but the inter-domain network performance may be less due to an
extra hop for every packet.
• As fewer LDC channels are consumed, it helps in creating more
VIO devices that require LDCs.
• This greatly helps customer configurations that have large
vnets in a Virtual Switch, especially when a customer doesn't
really care about inter-domain network performance.

Inter-vnet LDC Channels Explained
• LDoms CLI modification
– ldm add-vsw [default-vlan-id=<vid>] [pvid=<pvid>] [vid=<vid1,vid2,...>] [mac-
addr=<num>] [net-dev=<device>] [linkprop=phys-state] [mode=<mode>] [mtu=<mtu>]
[id=<switchid>] [inter-vnet-link=<on|off>] <vswitch_name> <ldom>
• The default setting is ON.
• This option is a Virtual Switch wide setting, that is enabling/disabling affects all
Vnets in a given Virtual Switch.
• Can be dynamically enabled/disabled without stopping the Guest domains.
– The Guest domains dynamically handle this change.
– ldm set-vsw [pvid=[<pvid>]] [vid=[<vid1,vid2,...>]] [mac-addr=<num>] [net-
dev=[<device>]] [mode=[<mode>]] [mtu=[<mtu>]] [linkprop=[phys-state]] [inter-vnet-
link=<on|off>] <vswitch_name>

Virtual Device Service Validation
• Support for Virtual Device Service Validation
• Enhances 'ldm add-vdsdev', 'ldm add-vswitch' and 'ldm bind'
commands to perform validation.
• Immediately validates the name and path for a specified
network device or virtual disk, greatly reducing the risk of
incorrectly configured I/O.
• This feature addresses the #1 cause of IO mis-configurations;
it gives the administrator immediate feedback if the
configuration is valid or invalid and why

Virtual Device Service Validation Example
• Examples:
# ldm add-vdsdev /bad/path bad_vol@primary-vds0
Path /bad/path is not valid on service domain primary
# ldm add-vdsdev -q /bad/path bad_vol@primary-vds0
# ldm list-bindings
…
VDS
NAME VOLUME OPTIONS MPGROUP DEVICE
…
bad_vol /bad/path
…
# ldm add-vswitch net-dev=bad-nic vsw1 primary
NIC bad-nic is not valid on service domain primary
# ldm add-vswitch -q net-dev=bad-nic vsw1 primary
…
VSW
NAME MAC NET-DEV ID DEVICE …
…
vsw1 00:14:4f:f8:02:08 bad-nic 1 switch@1 …
...

Oracle VM Server for SPARC 2.1
More Enhancements
• P2V Enhancements
• Bring more flexibility to quickly convert an existing SPARC server
running Oracle Solaris 8, 9 or 10 OS into a virtualized Oracle
Solaris 10 image to run on SPARC T-Series servers.
• Enhanced Management Information Base (MIB)
• Enables the SNMP MIB to use the latest Logical Domains
Manager XML interface, permitting 3rd party management
software to access the new features and resource properties.
• Lower-overhead, higher scalability networking for Oracle
Solaris 11 initial release
• Allows virtual network devices to use shared memory to exchange
network packets, enabling improved performance and scalability.

Oracle ILOM Key Functions
• Management Interfaces
• CLI, BUI, IPMI, SNMP
• Firmware Updates
• Remote Host Management
• Inventory and Component Management
• System Monitoring and Alert/Fault Management
• User Account Management
• Power Consumption Management

51
ILOM fdd diagnosis
• ILOM fault management uses 'fdd' diagnosis
• Currently supported on x86 Nehalem platforms
• All ILOM diagnosed problems will have a Message-
ID
• All ILOM diagnosed problems will have a
Knowledge Article

52
Fault Management
• Knowledge Articles in MOS
• ILOM fdd Diagnosis
• Faults and Alerts
• No ALOM Compatibility
• ILOM FMA Captive Shell
• Sideband Service Processor Network Connection
• New ILOM Fault Notification (SNMP Trap)
• ASR Support

53
No ALOM Compatibility
• ALOM functions not supported from ILOM CLI are
supported in Service or Escalation mode.

54
ILOM FMA Captive Shell
• Enter from ILOM CLI
• Display ILOM fdd diagnosed problems
• Display ereports from host FMA diagnosis
• Repair ILOM fdd diagnosed problems

55
ASR Support
• SPARC T4-4 will be supported by ASR (Automatic
Service Request) at release
• Continues use of sunHwTrapFaultDiagnosed SNMP
notification
• Telemetry for ILOM fdd diagnosis
• Supports platform and FRU identity
• Supports multi-suspect list

Side-band Management
• 3 Remote Management Communication Channels
• Out-of-band management = communicate with the SP over a
dedicated media (Ethernet/Serial)
• In-band management = communicate with the SP through
Oracle Solaris via agents
• Side-band management = communicate with the SP over a
shared media (the host’s data network interface)
• Side-band interface is Disabled by default (as shipped
from Factory)
• Can be enabled on any of the 4 on-board GigE Interfaces
• Configured from ILOM Web, CLI Interface or BIOS Setup Utility

Additional Resources
• T4-4 I/O Wiki

We encourage you to use the newly minted corporate tagline
“Hardware and Software, Engineered to Work Together.” at the end of
all your presentations. This message should replace any reference to
our previous corporate tagline “Hardware. Software. Complete.”

Oracle SPARC T4-4 Technical Overview

Recommended

Recommended

More Related Content

What's hot

What's hot (13)

Similar to Oracle SPARC T4-4 Technical Overview

Similar to Oracle SPARC T4-4 Technical Overview (20)

More from solarisyougood

More from solarisyougood (20)

Recently uploaded

Recently uploaded (20)

Oracle SPARC T4-4 Technical Overview

Editor's Notes