Sparc t4 2 system technical overview

<Insert Picture Here>
SPARC T4-2 System Technical Overview
Download this slide
http://ouo.io/v8AAK

The following is intended to outline our general product
direction. It is intended for information purposes only, and
may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality,
and should not be relied upon in making purchasing
decisions.
The development, release, and timing of any features or
functionality described for Oracle’s products remains at
the sole discretion of Oracle.

Agenda
• Oracle SPARC T4-2 Overview
• Oracle SPARC T4-2 Architecture
• Oracle SPARC T4-2 Components
• Oracle SPARC T4-2 Remote Management,
Oracle VM Server for SPARC

SPARC T4-2
Overview

SPARC T4-2 Mid-Range T-Series Server
2-Socket Enterprise-Class Data center Consolidation
& Back-Office Server
• Compute
• 2x SPARC T4 8-core 2.85GHz CPUs
• 32x DDR3 DIMMs, up to 512GB memory (16GB DIMMs)
• I/O and storage
• 6x 2.5” SAS-2 drives
• 10x PCIe2 slots (8 x8, 2 x4)
• 4x 1GbE ports
• 4x 10GbE XAUI ports (optional)
• Availability and management
• Redundant, hot-plug fans and power supplies
• Oracle Integrated Lights Out Manager Service Processor

SPARC T4-2
Target Workloads and Applications
• Web Application Server Consolidation
• Oracle Fusion Middleware (WebLogic)
• Oracle Cloud Computing
• OLTP Database Consolidation
• Security Applications
• Enterprise Applications – ERP/CRM
• Virtualization/Consolidation

SPARC T4-2 Design Overview
• 2-socket motherboard with SPARC T4 processors
• CPU: SPARC T4, 8 core, 64-thread V9 SPARC ISA
• SPARC binary compatibility preserved with prior generations of
SPARC systems
• 3RU Chassis with:
• 6x 2.5” Hot-pluggable drive bays
• 10x PCIe 2.0 Expansion slots
• 1x 10GbE QSFP Network Module (NM)
• 2x Hot-swappable and redundant PSUs w/ Power Sharing
• Hot-swappable and redundant fans
• Oracle ILOM Service Processor (AST 2200)

SPARC T4-2
Architecture

SPARC T4-2 CPU Configuration
• The SPARC T4-2 will support 2 CPUs only
• Features 2 SPARC T4 processors
• 8 cores per CPU, 8 threads per core → 128 threads
• Dual PCIExpress Gen2, Dual 10GbE ports
• Memory
• DDR3 DIMMs – 4GB/8GB/16GB sizes (32GB post-RR)
• 2 Memory Riser boards per CPU → 4 boards per system
• Next generation BoB (Buffers-on-board) between CPU and
Memory
• 2 BoBs per riser, 4 DIMMs per BoB
• 32 DIMMs per system → 512GB (@16GB DIMMs)

SPARC T4-2 Front view
6x 2.5” drives
SATA-based DVD+/-RW driveFront Video2x Front USB 2.0
Unrestricted opening for AirflowFront buttons and LEDs
RFID Tag
Wireless
Asset Tag

SPARC T4-2 Rear view
5x PCIe Slots
(0 to 4)
5x PCIe Slots
(5 to 9)
4x Gigabit
Ethernet
2x Rear
USB 2.0
2x Power
Inlets
Rear
Video ILOM
Serial
ILOM 10/100
Ethernet
Rear LEDs
and buttons
2x PSUs
4x 10GbE
NM

SPARC T4-2 System Block Diagram

SPARC T4-2 Top view

SPARC T4-2 Motherboard Layout
AST 2200
Service
Processor
(card not shown)
SPARC T4
CPU0
System TOD
Battery
SPARC T4
CPU1
PCIe Gen2
Switches
SAS Controller

T4-2 Chassis
• Main Features
• 6 Hot-Plug 2.5” SAS/SATA HDD/SSD bays
• 10 Low-profile PCIe expansion slots
• 2 Redundant Hot-Swap 2060W PSUs
• 1 SATA DVD +/- RW
• Dimensions
• 3U
• 5.11 in x 17.185 in x 28.81 in (H x W x D)
• 129.85 mm x 436.55 mm x 732 mm (H x W x D – metric)
• Max Weight = 36.8 kg (80 lbs)
• Depth with PSUs Ejector Handles = 29.6 in (752.35 mm)

SPARC T4-2
Components

SPARC T4-2 Memory Architecture
• Each SPARC T4 CPU
contains 2 memory
controllers
• MCU0 and MCU1
• Each controller has
interfaces to 2 Buffers
on Board (BoBs)
• 2 DDR3 channels
• 2 DIMMs per channel
• 16 DIMMs per CPU
• Memory bus speed
limited to 1066 MHz for
all platforms

SPARC T4-2 Memory Population
General Rules
• Rules
• Minimum capacity is 64GB
• Maximum capacity is 512GB (1.024TB post-RR)
• W5C population rules – half-populated (8 DIMMs per node) or
fully-populated (16 DIMMs per node)
• Recommend minimum of 16 DIMMs per system which provides
full memory bandwidth

SPARC T4-2 Memory Configuration Rules
• 4 memory risers with 8 DIMM slots per riser
• Systems are preloaded with 4 memory risers
• Minimum of 4 DIMMs per riser must be installed and each
channel must be filled by at least one DIMM
• Minimum 2 memory kits (4 DIMMs slots) must be
installed/configured per riser
• Memory can be added in quantities of 2, 4 or 8 memory kits
• Best practice install is 8 memory kits
• Unpopulated DIMM slots must have a DIMM filler
• Memory risers are inaccessible in event of CPU failure

SPARC T4-2 Memory Population
Populating Risers
Each CPU socket is connected to 2 memory riser slots (MR0
and MR1)
T4-2 Systems support a maximum of 4 memory risers
Performance-oriented configurations should be configured
with 2 memory risers per CPU

SPARC T4-2 Memory Configuration
Populating DIMMs on a Riser
• DIMMs must be populated in matching pairs of identical
DIMMs (Lock-Step Pairs = D0/D4, D1/D5, D2/D6 and
D3/D7)
• DIMM slots are color coded. The memory population
order is represented on the service label inside the
chassis top cover
• Within a riser, DIMMs should be populated in that order:
• D0/D4 (blue slots)
• D2/D6 (white slots)
• D1/D5 (black slots)
• D3/D7 (green slots)

T4-2 Memory Population

T4-2 Memory Configuration

SPARC T4-2 Memory Fillers
• Fillers reduce fan power and optimize system cooling
• Use memory DIMM fillers in unpopulated DIMM slots
Memory DIMM Fillers

SPARC T4-2 PCI Express Expansion Slots
• SPARC T4-2 has 10 low-profile PCIe slots
• Slots 0 to 5 are connected to Switch 0
• Slots 6 to 9 are connected to Switch 1
• Slots 4 and 5 are x4 electrical (x8 mechanical)
• All other slots are connected to the Switches using 8 lanes
• Even slots are connected to CMP0, odd slots to CMP1
• Slots 3 and 8 are x8 electrical (x16 mechanical)

SPARC T4-2 PCI Express Expansion Slots
• PCIe Slots Population Considerations
• Consult T4-2 product notes or T4 Internal FAQ
• PCIe Filler Panel
• Any empty PCI Express slot requires a PCIe Filler Panel

SPARC T4-2 PCI Express Expansion Slot

SPARC T4-2 Host Network Interfaces
• 4 onboard Gigabit Ethernet ports located rear of the
server (RJ45 connectors)
• Controller Intel 82576
• 10/100/1000Mb/s full- and half-duplex operation
• IEEE 802.3ab Auto-Negotiation for speed, duplex, and
flow control

SPARC T4-2 Host Network Interfaces
• Ethernet 0 and 1 dual NIC is
directly connected to CMP0
via Switch1
• Ethernet 2 and 3 dual NIC is
directly connected to CMP1
through Switch1

SPARC T4-2 Internal Storage - HDDs
• 6 bays 2.5”, Hot-Plug/Swap, Front Accessible
• SAS Gen 2 (6 Gbps)
• 2 mini-SAS cables connect the disk backplane to
either
• RAID HBA in one of the PCIe slots
• PCIe option card required for RAID5
• RAID 0 (striping), RAID 1 (mirroring) using 'raidctl’ and
RAID 1E (enhanced mirroring)
• SAS HDDs
• 300 GB 10,000 RPM SAS2
• 600 GB 10,000 RPM SAS2

SPARC T4-2 Internal Storage
HDD Status LEDs
Hard Disk Drive LEDs
(1) Ready to Remove (Blue)
(2) Service Required (Amber)
(3) OK/Activity (Green)

SPARC T4-2 Internal Storage
Miscellaneous
• Internal Solid-State Disk (SSD)
• 100GB, 300GB
• Internal DVD
• One slot-loading DVD+/-RW drive in front
• Located below the drive bays
• Internal USB
• Not supported in SPARC T4-2, but useable

SPARC T4-2 I/O Ports
• Ports and Connectors
• 2 VGA ports, DB-15 (1 Front and 1 Rear)
• 5 USB 2.0 ports, Type A (2 Front, 2 Rear, 1 Internal)
• 4 Gigabit Ethernet ports, RJ-45 (Rear)
• 1 Service Processor 10/100 Ethernet port, RJ-45 (Rear)
• 1 Service Processor RS-232 Serial port, RJ-45 (Rear)
• 2 AC Power inlets120/240V, IEC-320 C-14 (Lower Left Rear)
• 10 PCIe Expansion slots
• 1 Network Module (NM)
• VGA Ports
• Both front and rear video ports are active at all times
• Only rear video port has monitor identification. Connect monitor
to rear port if identification is necessary for OS operations

SPARC T4-2 Switches and Buttons
• Switches and Buttons
• 1 Power button
• 2 Locate buttons (1 Front and 1 Rear)
• Service Processor Password Reset button (Hidden Pin Hole in
rear)
• Chassis Interlock/Intrusion magnetic switch (Top of chassis,
activated by top cover removal)
• Internal Buttons
• Main Fault-Remind button
• Fault-Remind button on each riser module

SPARC T4-2 Chassis LEDs
• Front and rear Locate LEDs (White)
• Front and rear System Fault LEDs (Amber)
• Front and rear Power/OK LEDs (Green)
• SP OK/Fault LED (bi-color Amber/Green)
• 6 Front facing Amber Fault LEDs (All Amber)
• Chassis Alert
• Fan Fault
• Processor Fault
• Memory Fault (DIMM)
• Power Supply Fault
• Over Temperature Warning

SPARC T4-2 Fault Tracing
• Fault-Remind buttons help identify or locate a faulty
component even after the system has been disconnected
from the power source
• Each Fault-Remind button will lit different sets of
components
• Main Fault-Remind button
• Processor Shift, Memory Riser
• Memory Riser Fault-Remind button
• Shift DIMMs

SPARC T4-2 Chassis Cooling
• Front to back cooling
• 2 independent Cooling compartments
• Compartment 1: Drive bays and PSUs are cooled by airflow
suction from the AC-DC PSUs fans
• Compartment 2: Motherboard, memory risers and I/O cards
cooled by six 92mm fans controlled by ILOM
• Air is pressurized in the main compartment for front to back
cooling
• Divided into 3 independently controlled zones (Z0 – Z2)
• Higher efficiency (each zone operates at its highest efficiency)
• Less noise (unrestricted airflow for motherboard)

Cooling Zones
• Chassis Cooling Zones
• 6 Fans pull air in
• Cools Motherboard
• Subdivided in 3 zones
• PS Cooling Zone
• PSU Fans pull air out
• Cools Drives, Fmods,
PSUs
PCIe Slots
Chassis Zone Z0
Chassis Zone Z1
Chassis Zone Z2
PS Zone Z3 PS Cooling Zone
Chassis Cooling
Zone
CPUs Memory Risers
Power Supplies Drives
Rear Fan Row
Front Fan Row
Air Divider

Front Fans
• 6 Redundant Hot-Swap fans
• Vibration isolated from chassis
• 92 mm high performance fans
• Rated 4.3 A
• 2 rows (front and rear) of 3 fans for redundancy
• The 3 fans in the rear row must be populated for the system to
turn ON
• If replacing rear row fans while system is turned on, there is a 30-
second replacement time limit

PSU Fans
• Each PSU incorporates 2 columns of 2 redundant fans
• Replace the power supply in the event of a failure of a fan

Fillers
• Airflow and Fillers
• Each drive bay must be populated with a storage device or a
drive filler
• Each memory riser slot must be populated with a memory riser or
a memory riser filler
• Each DIMM slot to be populated with DIMM or DIMM filler
• Each PCIe slot must be populated with a PCIe card or have a slot
filler (perforated back plate)
• All necessary fillers ship with the system

SPARC T4-2 Power Supplies
• Each PSU rated 2060W / 1030W continuous output
• Main output 12V @165 Amps at high line (200-240V)
• Standby output 3.3V @10 Amps
• Per-cord ratings
• 10A at 220-240V (2200W max)
• 11A at 200-219V (2200W max,
Japan & North America only) AC Present
Service
Required
OK

SPARC T4-2 Power
• SPARC T4-2 is supplied with 2 PSUs
• In case of PSU failure, will run with single PSU
• Second PSU must be present, even if failed, in second slot to
prevent air flow recirculation and for EMI containment
• If a PSU is missing, the Front and Rear amber Service LEDs as
well as the “PS REAR” LED on the front bezel will be lit

Power Availability
Redundant Over-subscription
• System design uses “right sized” PSUs for efficiency
• Prefer PSU sized for 2/3 of max system load, plus 50% surge
• A high line PSU can support max-power operation
• Even future components are likely to draw <2200W input power

SPARC T4-2 Rack Mounting Options
• SPARC T4-2 is supported in the following Sun Rack
cabinets
• Sun Rack II 1042
• Sun Rack II 1242
• Racking included in ATO base chassis
• Cable Management Arm
• Tool-less Rack Mount Slide Rail kit

SPARC T4-2 Environmentals
• Acoustic Noise: 61.5 dBa operating max
• Operating Temperature: 5C to 35C (41F to 95F)
• Nonoperating Temp.: -40C to 65C (-40F to 149F)
• Operating Humidity: 10% to 90%, noncondensing
• Nonoperating Humidity: 93%, noncondensing
• Cooling: 4361.9 BTU/hr, 230 cfm max

SPARC T4-2
Remote Management,
Oracle VM server for SPARC

SPARC T4-2 Service Processor
• The Oracle ILOM SP (ILOM) is integrated
• ARM9-based AST2200 from Aspeed with 128M RAM and
32M flash
• ILOM controls all LEDs and Fans
• Serial port connected to ILOM by default
• On-board Graphics Specs
• 8 MB Video Memory
• 1280x1024 Max resolution

Oracle ILOM Key Functions
• Management Interfaces
• CLI, BUI, IPMI, SNMP
• Firmware Updates
• Remote Host Management
• Inventory and Component Management
• System Monitoring and Alert/Fault Management
• Thermal and Fan Control
• User Account Management
• Power Consumption Management

Side-band Management
• 3 Remote Management Communication Channels
• Out-of-band management = communicate with the SP over a
dedicated media (Ethernet/Serial)
• In-band management = communicate with the SP through Oracle
Solaris via agents
• Side-band management = communicate with the SP over a
shared media (the host’s data network interface)
• Side-band interface is Disabled by default (as shipped
from Factory)
• Can be enabled on any of the 4 on-board GigE Interfaces
• Configured from ILOM Web, CLI Interface or BIOS Setup Utility

© 2011 Oracle Corporation
SPARC T4-2 System RAS Overview
• Designed to minimize part count and operating
temperature to enhance reliability
• End-to-end data protection detecting and correcting
errors throughout server – ECC everywhere
• Processor and Memory protection
• CPU core and thread off-lining
• Memory with ECC, x4/x8 DRAM Extended ECC, page
retirement, and lane failover
• Major components redundant & hot-replaceable
• Fan, Power Supply, and internal disks
• RAID capability for internal disks

T4-2 Software/Firmware Block Diagram
Linux KernelLinux Kernel
Guest MgrGuest Mgr
UBoot/DiagsUBoot/Diags
Service Processor
ILOM
Host ConfigHost Config
HypervisorHypervisor
- Environmentals
- Fault Management
- LED Control
- SP Diags
- DFRUID
- Plat HW Svc
- FMA ETM
- IPMI
- CLIs
- Logs
- SNMP
- FMA Support
- PowerOn/Off
- FERG
CPU (AST2200)CPU (AST2200)
CPUCPU MemoryMemory IOIO
Host
LDC Channels
- Host Config
- Machine Description
- Hypervisor
- OBP
- POST
Host Flash
OBPOBPOBPOBP
Solaris
S10U10
Solaris
S10U10
sun4vsun4v
Solaris 11
Express
Solaris 11
Express
sun4vsun4v
OBPOBP
POSTPOST
System
Domain
System
Domain
LDOMS
Manager
sun4vsun4v
Host Data Flash
- OBP NVRAM/POST/SC config vars
- ASRDB -LDOMS config
- Console Log - SER log – TOD data
Host Data Flash
- OBP NVRAM/POST/SC config vars
- ASRDB -LDOMS config
- Console Log - SER log – TOD data
Platform Hardware
FPGAFPGA
- Kernel
- FMA Components
- Platform Drivers
- Kernel
- FMA Components
- Platform Drivers
- S10U10
- FMA Components
- Platform Drivers

© 2011 Oracle Corporation Page 54
Oracle Solaris and SPARC Virtualization
Better Resource Utilization for a More Efficient Data center
Dynamic Domains Oracle VM Server
for SPARC
M-Series T-Series
App App
Oracle Solaris
Containers
Oracle Solaris
DW DB
Domain A
Domain B
OLTP DB
OLTP DB
App
App
Domain A
Domain B
Domain C
Web
Web DB App Web
Web
Web

Oracle VM Server for SPARC 2.1
Customers benefit from increased application service level
• New Hardware Support
• SPARC T4 servers with Oracle Solaris 10 8/11; Solaris 11 post-Release
• Secure live migration
• Dynamic resource management (DRM) between domains
• Integrated Dynamic Reconfiguration (DR) of cryptographic
units and virtual CPUs.
• Increased maximum number of virtual networks per domain
• Support for virtual device service validation
• Lower-overhead, higher scalability networking for Oracle
Solaris 11 initial release
• Enhanced Management Information Base (MIB)
• Physical to virtual (P2V) enhancements

Secure Live Migration
• Live migration available on
SPARC T-Series systems
• SPARC T4
• SPARC T3
• UltraSPARC T2 Plus
• UltraSPARC T2
• On-chip crypto accelerators
deliver secure, wire speed
encryption for live migration
• No additional hardware required
• Eliminates requirement for
dedicated network
• More secure, more flexible
VM
External Shared Storage
SPARC T-Series servers
Oracle VM Server Pool
VM Secure Live Migration (SSL)VM VM
Eliminates Application Downtime

Live Migration Requirements
• Source System
• Guest domain with virtual I/O devices only, running Solaris 10 9/10
• Requires Logical Domains Manager 2.1 & updated firmware (i.e.
7.4 or 8.1)
• Can not have multi-pathed disks (IPMP for networks is ok)
• Power Management in “performance” mode (the default)
• Target System
• Must have enough resources (cpu, mem)
• Must have appropriate VIO services (vds, vsw, vcc)
• Must be able to provide required VIO devices (vdisk, vnet)
• Must be cpu-compatible
• Same processor type (e.g. SPARC T4), same clock frequency

Perform Live Domain Migration
• Uses the same CLI and XML interfaces as in prior
releases
• Also from Oracle Enterprise Manager Ops Center
• CLI example
– ldm migrate [-f] [-n] [-p <password_file>] <source-ldom>
[<user>@]<target-host>[:<target-ldom>]
• -n : dry-run option
• -f : force
• -p : specify password file for non-interactive migration
• Cancel an On-Going Migration
– ldm cancel-operation migration <ldom>

Live Migration Best Practices
• There is no specific requirement on the number of CPUs
in the control domains. However our experience shows
that > 8 vCPUs is best. We recommend 16 or more
vCPUs in order to minimize the suspend time as well as
the overall migration time.
• Workloads that heavily modify memory pages will have
longer migration times
• Be sure to add the crypto units for best migration
performance.
• Review the documentation, especially the Admin Guide
for planning live migration
• White paper on best practices to be published soon

Resource Management Improvements
• Dynamic resource management (DRM) between domains
• Dynamic CPU movement is based on the priority property of each
domain's DRM policy.
• Ensures that domains running the most important workloads get
priority for CPU access over domains with less critical workloads
• Integrated Dynamic Reconfiguration (DR) of Cryptographic
units and virtual CPUs
• Automatically remove crypto unit when the final vCPU of a core is to
be removed.
• Simplify operations and ensure consistent performance.

Increased Maximum Number of Virtual
Networks Per Domain
• Introduces an option to dynamically disable the allocation
of inter-vnet Logical Domain Channels (LDC)
• The number of LDCs is limited by hardware.
• Inter-domain communication continues to work exactly as before,
but the inter-domain network performance may be less due to an
extra hop for every packet.
• As fewer LDC channels are consumed, it helps in creating more
VIO devices that require LDCs.
• This greatly helps customer configurations that have large
vnets in a Virtual Switch, especially when a customer doesn't
really care about inter-domain network performance.

Inter-vnet LDC Channels Explained
• LDoms CLI modification
– ldm add-vsw [default-vlan-id=<vid>] [pvid=<pvid>] [vid=<vid1,vid2,...>] [mac-
addr=<num>] [net-dev=<device>] [linkprop=phys-state] [mode=<mode>] [mtu=<mtu>]
[id=<switchid>] [inter-vnet-link=<on|off>] <vswitch_name> <ldom>
• The default setting is ON.
• This option is a Virtual Switch wide setting, that is enabling/disabling affects all
Vnets in a given Virtual Switch.
• Can be dynamically enabled/disabled without stopping the Guest domains.
– The Guest domains dynamically handle this change.
– ldm set-vsw [pvid=[<pvid>]] [vid=[<vid1,vid2,...>]] [mac-addr=<num>] [net-
dev=[<device>]] [mode=[<mode>]] [mtu=[<mtu>]] [linkprop=[phys-state]] [inter-vnet-
link=<on|off>] <vswitch_name>

Virtual Device Service Validation
• Support for Virtual Device Service Validation
• Enhances 'ldm add-vdsdev', 'ldm add-vswitch' and 'ldm bind'
commands to perform validation.
• Immediately validates the name and path for a specified
network device or virtual disk, greatly reducing the risk of
incorrectly configured I/O.
• This feature addresses the #1 cause of IO mis-configurations;
it gives the administrator immediate feedback if the
configuration is valid or invalid and why

Virtual Device Service Validation Example
• Examples:
# ldm add-vdsdev /bad/path bad_vol@primary-vds0
Path /bad/path is not valid on service domain primary
# ldm add-vdsdev -q /bad/path bad_vol@primary-vds0
# ldm list-bindings
…
VDS
NAME VOLUME OPTIONS MPGROUP DEVICE
…
bad_vol /bad/path
…
# ldm add-vswitch net-dev=bad-nic vsw1 primary
NIC bad-nic is not valid on service domain primary
# ldm add-vswitch -q net-dev=bad-nic vsw1 primary
…
VSW
NAME MAC NET-DEV ID DEVICE …
…
vsw1 00:14:4f:f8:02:08 bad-nic 1 switch@1 …
...

Oracle VM Server for SPARC 2.1
More Enhancements
• P2V Enhancements
• Bring more flexibility to quickly convert an existing SPARC server
running Oracle Solaris 8, 9 or 10 OS into a virtualized Oracle
Solaris 10 image to run on SPARC T-Series servers.
• Enhanced Management Information Base (MIB)
• Enables the SNMP MIB to use the latest Logical Domains
Manager XML interface, permitting 3rd party management
software to access the new features and resource properties.
• Lower-overhead, higher scalability networking for Oracle
Solaris 11 initial release
• Allows virtual network devices to use shared memory to exchange
network packets, enabling improved performance and scalability.

We encourage you to use the newly minted corporate tagline
“Hardware and Software, Engineered to Work Together.” at the end of
all your presentations. This message should replace any reference to
our previous corporate tagline “Hardware. Software. Complete.”

Sparc t4 2 system technical overview

Recommended

Recommended

More Related Content

What's hot

What's hot (8)

Viewers also liked

Viewers also liked (8)

Similar to Sparc t4 2 system technical overview

Similar to Sparc t4 2 system technical overview (20)

More from solarisyougood

More from solarisyougood (20)

Recently uploaded

Recently uploaded (20)

Sparc t4 2 system technical overview

Editor's Notes