Uploaded byxKinAnx
PDF, PPTX2,916 views

Sun sparc enterprise t5440 server technical presentation

The document provides an agenda for a training course on Sun SPARC Enterprise T5440 rack servers. The agenda includes an introduction, comparison of UltraSPARC T2 and T2 Plus processors, overview of T5440 server features and architecture, memory, networking, I/O expansion, disks, fans, power supplies, Solaris, ILOM 2.0, LDOMs, CRUs/FRUs, tuning and performance, tools and references. It also notes that the information is confidential to Sun Microsystems.

Embed presentation

Download as PDF, PPTX
Sun SPARC Enterprise T5440 Rack Servers Train-the-Trainer Course Presenter Sun Microsystems, Inc. Sun Confidential CDA Required 1
NDA Notification This presentation contains information related to projects that are currently in planning and/or development stages. This information represents the current intentions of Sun Microsystems, Inc. However, all aspects of these projects, including, but not limited to funding, availability, shipping dates, configurations, capacities, performance, and all other characteristics are subject to change and/or cancellation without notice. This material is confidential to Sun Microsystems and should be disclosed to non- employees only under the terms of an executed confidential-disclosure agreement. Distribution of this material to Sun Field Representatives should be done through the GSO CDA process. Under no circumstances should these roadmaps be given directly to a Sun Field Representative. The information contained herein is considered privileged, confidential and protected from disclosure. If you are not the intended recipient or their employee/agent responsible for its delivery, you are requested to refrain from any duplication or distribution of this message or its contents, in respect of the rights of the designated recipient and sender and to delete it from your computer. Sun Internal and Approved Partners Only 2
Agenda • Introduction • Comparision of the UltraSPARC T2 to UltraSPARC T2 Plus • Sun SPARC Enterprise T5440 server > Overview of Features > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 2.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 3
Industry's Most Highly Threaded Servers Maximum Threading = Higher Throughput, Greater Energy & Space Efficiency T5440 Up to 256 Threads T5140/T5240 Up to 128 Threads T5120/T5220 Up to 64 Threads SB6320 US T2 blade Sun Internal and Approved Partners Only 4
UltraSPARC T2: • Up to 8 cores @1.2GHz or 1.4GHz TrueSystem On a Chip • Up to 64 threads per CPU • Up to 16 FB-DIMMs, 4 memory controllers FB DIMM FB DIMM FB DIMM FB DIMM > Up to 64GB memory (4GB DIMMs) > 2.5x memory BW of UltraSPARC T1 FB DIMM FB DIMM FB DIMM FB DIMM • 8 x fully pipelined Floating Point units / core, 1 per core MCU MCU MCU MCU • Dual 10Gbit Ethernet and PCI-E integrated onto chip L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$ • 4MB L2 (8 banks) 16 way associative Full Cross Bar • Enhanced MAU/Security co-processor C0 C1 FPU FPU C2 FPU C3 FPU C4 FPU C5 FPU C6 FPU C7 FPU per core MAU MAU MAU MAU MAU MAU MAU MAU > DES, 3DES, AES, RC4, SHA1, DMA Sub-System SHA256, MD5, RSA to 2048 key, ECC,CRC32 NIU Sys I/F • Advanced Power saving features (E-net+) Buffer Switch Core PCI-E • 65nm process technology Power ~95W <1.5W/thread x8 @2.5GHz 2x 10GE Ethernet 4GBytes/s bi-directional Sun Internal and Approved Partners Only 5
UltraSPARC T2 Processor SPC SPARCv9 CMT Core L2 Data L2 Data Bank 0 Bank 4 CCX Core-Cache Crossbar L2B0 SPARC SPARC SPARC SPARC L2B4 L2$ Level 2 Cache Bank L2 Data Core 0 Core 1 Core 5 Core 4 L2 Data CLC Coherence & Link Controller Bank 1 Bank 5 MCU Memory Control Unit L2B1 L2B5 SIU System Interface Unit L2 L2 L2 L2 MCU0 MCU2 NCU Non-cacheable Unit NCU TAG0 TAG1 TAG5 TAG4 FSR MAC Media Access Controller EFU MCU1 MCU3 DMU Data Management Unit SIO SII L2B3 CCX L2B7 FSR CCU PEU PCI Express Unit L2 Data L2 Data Bank 3 Bank 7 L2 L2 L2 L2 L2B2 L2B6 TAG2 TAG3 TAG7 TAG6 L2 Data L2 Data Bank 2 Bank 6 SPARC SPARC SPARC SPARC DMU Core 2 Core 3 Core 7 Core 6 RDP TDS PEU RTX PSR ESR FSR MAC For a video tour with Rick Hetherington, go to http://www.youtube.com/watch?v=B8DNexYcsa4 Sun Internal and Approved Partners Only 6
UltraSPARC T2 Plus: • Per Socket Multi-Socket US T2 > Up to 8 cores @1.2GHz or Each Coherency Link 6.4GBytes/s per direction, 12.8GB/s total 1.4GHz 4 x Coherency links give Snoop bandwidth of 51.2GB/s > Up to 64 Threads per socket 4x Coherency > 4MB L2$ 8 banks x 16 Way SA Links > Up to 8 FPU's, 1 per core FB-DIMM FB-DIMM FB-DIMM FB-DIMM > Up to 8 Crypto cores, 1 per core FB-DIMM FB-DIMM FB-DIMM FB-DIMM > DES, 3DES, AES, RC4, SHA1, SHA256, MD5, RSA to 2048 key, ECC,CRC32 MCU MCU > 2 memory Controllers, 16 DIMMS CU CU CU CU > 25Read+13WriteGB/s Memory BW L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$ > One x8 PCIe interface Full Cross Bar • Per system CO C1 C2 C3 C4 C5 C6 C7 > 2 sockets glue-less FPU MAU FPU MAU FPU MAU FPU MAU FPU MAU FPU MAU FPU MAU FPU MAU > 4 sockets using UST2 Plus XBR > 16 or 32 PCIe lanes on 2 or 4 System I/F PCI-Express Socket systems • 65nm process technology SSI x8 @2.5GHz Bus Power ~105W 4GBytes/s <1.5W/thread bi-directional Sun Internal and Approved Partners Only 7
UltraSPARC T2 Plus Processor SPC SPARCv9 CMT Core CCX Core-Cache Crossbar L2$ Level 2 Cache Bank CLC Coherence & Link Controller MCU Memory Control Unit SIU System Interface Unit NCU Non-cacheable Unit NCX Non-cacheable Crossbar DMU Data Management Unit PEU PCI Express Unit Sun Internal and Approved Partners Only 8
CMT Processor Comparison UltraSPARC T1 UltraSPARC T2 UltraSPARC T2 Plus Technology 90nm 65nm 65nm # of Cores 8 8 8 # of Threads 32 64 64 # of Sockets 1 1 1–4 Core Frequency (Ghz) 1 – 1.4 1.2 – 1.4 1.2 – 1.4 Execution pipelines per core 1 2 2 Instructions per cycle per pipe 1 1 1 FPUs per chip 1 8 8 I$ size (per core) 16KB 16KB 16KB D$ size (per core) 8KB 8KB 8KB L2$ size/Set Association 3MB/12-Way 4MB/16-Way 4MB/16-Way Memory Type DDR2 FBDIMM-1 FBDIMM-1 Memory Bandwidth (GB/s) 21.3 42READ + 21WRITE 25READ + 13WRITE Coherence Bandwdth (GB/s) - - 4 x (6.4READ +6.4WRITE) IO Bus JBUS x8 PCIe x8 PCIe IO Bandwidth (GB/s) 2.5 2+2 2+2 On-chip Networking - 2 x 10GE - Crypto Acceleration RSA RSA, RC4, DES, 3DES, AES, SHA1-256, MD5 Power (Watts, Watts/thread) 72, 2.25 95, 1.5 105, 1.6 * Sun Internal and Approved Partners Only 9
UltraSPARC T2 Plus 2-Socket System Coherency Links Dual Channel Dual Channel Dual Channel Dual Channel FBDIMM FBDIMM FBDIMM FBDIMM Memory Controller Memory Controller Memory Controller Memory Controller Coherence Coherence Coherence Coherence Coherence Coherence Coherence Coherence Unit Unit Unit Unit Unit Unit Unit Unit UltraSPARC T2 Plus Cores, Crossbar, L2$ UltraSPARC T2 Plus Cores, Crossbar, L2$ (8 cores, 64 threads, 4MB L2$) (8 cores, 64 threads, 4MB L2$) PCI-Express NCU, DMU NCX PCI-Express NCU, DMU NCX System IO (Network, Disk, etc.) Sun Internal and Approved Partners Only 10
UltraSPARC T2 Plus 4-Socket System 6.4GB/s 6.4GB/s Dual Channel FBDIMM Dual Channel US T2 Plus US T2 Plus FBDIMM PCI-Express PCI-Express 8 Cores 6.4GB/s 6.4GB/s 8 Cores 6.4GB/s 6.4GB/s 64 Threads Coherence 64 Threads Dual Channel 4MB L2$ 4MB L2$ Coherence(4) Hubs FBDIMM Dual Channel FBDIMM Coherence Coherence UltraSPARC Hubs (4) Hubs (4) HubsPlus T2 (4) 6.4GB/s XBR 6.4GB/s 44 ports/hub ports/hub 6.4GB/s 4 ports/hub 6.4GB/s Dual Channel inin each dir each dir FBDIMM Dual Channel in each dir FBDIMM PCI-Express US T2 Plus VF 4 ports/hub US T2 Plus PCI-Express 6.4GB/s per port 6.4GB/s 6.4GB/s 8 Cores 8 Cores in each dir 6.4GB/s 8 Cores 6.4GB/s 64 Threads 8 Threads/Core 64 Threads Dual Channel FBDIMM 4MB L2$ 4MB L2$ Dual Channel 4MB L2$ FBDIMM 6.4GB/s 6.4GB/s Sun Internal and Approved Partners Only 11
US T2 Plus XBR Coherency Hub Chips... • 4 XBR chips in a system • XBR routes non-cacheable PIOs to the • Each operates in its own appropriate IO device Coherency Plane(CP) > Forwarded to the Zambezi that owns the > A region of addressable space, unique address space for processing to that plane • DMA reads and writes are done > Each XBR only tracks addresses in its own space through cachable memory space just > Serializes accesses to coherent loads, like processor initiated memory stores and writebacks (TSO model) accesses > 40 bit Physical Address used • CP links are based on an FB-DIMM > VF Snoop links to pass transactions physical interface, but a Sun protocol • Cacheable transactions > 6.4GBps/Direction/Port – Most Memory transactions > 51.2GB/s bandwidth total • Non-Cacheable transactions > CP links are tolerant to a single lane failure – IO, Non-cachable memory, (14 lanes per link) Interrupts etc. Coherency Plane > CRC and replay features in the protocol Mem Interleave by Designator allow no-stop recovery Coherency Plane Cachable/ Non-Cachable Coherency plane Designator (512B/1GB) > Memory structures are Parity protected 39 31 30 40 Bit Physical Address 10 9 8 7 0 Sun Internal and Approved Partners Only 12
UltraSPARC T2/Plus CPU Core Block Diagram • IFU – Instruction Fetch Unit > 16 KB I$, 32B lines, 8-way SA TLU IFU > 64-entry fully-associative ITLB • EXU0/1 – Integer Execution Units > 4 threads share each unit > Executes one integer instruction/cycle EXU0 EXU1 • LSU – Load/Store Unit > 8KB D$, 16B lines, 4-way SA MMU/ > 128-entry fully-associative DTLB HWTW • FGU – Floating/Graphics Unit • SPU – Stream Processing Unit > Cryptographic acceleration SPU FGU LSU • TLU – Trap Logic Unit > Updates machine state, handles exceptions and interrupts • MMU – Memory Management Unit > Hardware tablewalk (HWTW) Gasket – TLB miss handled by hardware > 8KB, 64KB, 4MB, 256MB pages xbar/L2 Sun Internal and Approved Partners Only 13
Floating Point Unit • Each Core has its own Floating Point Unit. This removes completely the 2% FP restriction of T1. • Only a 6 cycle penalty to the FPU > T1 required upwards of 28 cycles • Fully-pipelined (except divide/sqrt) Data > Divide/sqrt in parallel with add or multiply operations of other threads • Full VIS 2.0 implementation in hardware, no emulation • FPU performs integer multiply, divide, population count • Multiplier incorporates enhancements for offloading modular arithmetic operations to the Crypto Unit ● ● Sun Internal and Approved Partners Only 14
UltraSPARC T2 Plus Crypto Performance Absolute maximums, measured by engineering using micro-benchmarks Application throughputs will be lower than these numbers Iterations/ Block Cycles/ Function Comment Block (bytes) Block GB/S DES 2 rounds/cycle 16 rounds 8 8/9+ 12.0 3DES 2 rounds/cycle 48 rounds 8 25/24 4.0 RC4 3 overlapped cycles 1 1 37683 12.0 Data into 44 words with UltraSPARC T2 Plus AES-128K 4 words - - 32 - AES-192K 6 words into 52 words - - 37 - New Ciphers AES-256K 8 words into 60 words - - 41 - AES-128 2 column over two cycles 10 rounds 16 30/30 6.36 AES-192 2 column over two cycles 12 rounds 16 36/36 5.28 AES-256 2 column over two cycles 14 rounds 16 42/42 4.56 SHA-1 2 cycles per step 80 steps 64 160/160 4.80 SHA-256 2 cycles per step 64 steps 64 128/128 6.00 MD5 2 cycles per step 64 steps 64 128/128 6.00 CRC32c 32 bits per cycle 1 4 1 48.00 Wire Speed for UltraSPARC T2 Plus is around 5 GB/S 16,000 new SSL sessions/sec using 1024 Asymmetric Key Exchange Export restrictions apply to some of the ciphers, requires installation of the Crypto Enabling patch to use. Purchase a CD which contains the patch. Affects IPsec related ciphers, SSL typeSun Internal and Approved Partners Only acceleration is unaffected and does not need enabling. 15
FB-DIMM Memory Interface Feature UltraSPARC T1 UltraSPARC T2 UltraSPARC T2 Plus Require Pins 1000 440 208 Available Bandwidth 23GBytes/second 50Gbytes/second 37.5Gbytes/second/socket Up to 64x DIMM slots Up to 32x DIMM slots/socket Memory Capacity up to 16x DIMM slots T5x20 uses 16 DIMM slots T5x40 uses 8/16 DIMM slots Data Memory Controllers 4 4 2 Memory Latency 96ns 126ns 126ns DIMM speeds 400MHz 667MHz 667MHz DIMM Power 7watts/DIMM 12.5Watts/DIMM 12.5Watts/DIMM ● ● Sun Internal and Approved Partners Only 16
10/100/1000/10GbE Network Interfaces • Uses the Neptune Chip on the Motherboard which provides > Two RJ-45 ports dedicated for 10/100/1000 > Two RJ-45 ports capable of 10/100/1000 – Fitting a XAUI adapter disables the one of these ports per XAUI • Same 10 GbE XAUI cards as T5x20 server • 16 Rx and 24 Tx DMA channels for virtualization – Integrated NIU in T2 has 16 Tx channels Data – Raw UDP Tx performance on Neptune > T2 NIU – T2 processor network typically outperforms Neptune on most network benchmarks • Packet classification > Rx data can be HW classified at layer 2 ,3 and 4 – Put into the separate Rx DMA buffers by classification rules • Same 'nxge' device driver as T2 NIU. • Performance estimates, testing/tuning ongoing – 1 port 10GbE TX/RX = 9.3 Gbps with CPUs still 65-75% idle ● – 2 port 10GbE TX/RX = 12-19Gbps (better if dedicate RX/TX ports) ● Sun Internal and Approved Partners Only 17
Agenda • Introduction • Comparision of the UltraSPARC T2 to UltraSPARC VF • Sun SPARC Enterprise T5x40 servers > Overview of Features > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 2.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 18
SPARC Enterprise Compute Density • 4RU chassis T5440 • 4-sockets – UltraSPARC T2 Plus Quad Socket US T2 Plus > Up to 32 cores @ 1.2GHz or 1.4Ghz > Up to 256 threads • Up to 64 FBDIMM slots > 512 GB of memory (using 8GB DIMMS) Reliable • 4 Hot-Plug 2.5” SAS disks (Raid 0,1) • DVD-RW • 4 x Redundant(2+2), hot-swappable PSUs ● Up to 256 threads / 512GB memory • 4 x Redundant, hot-swappable Fans • 2N PSU Redundancy, 1900 watts typical in a robust, redundant chassis consumption ● Taking CMT to the Mid-Range Expandable • 4 x Processor+Memory card pairs ● LDOMs consolidation dream box • 8 x8 PCIe expansion slots (2 x XAUI) • 4 Gigabit Ethernet (or up to 2 x 10GbE and 2 x1GbE) • 4 x USB 2.0 Sun Internal and Approved Partners Only 19
T5440 Standard Configurations (PTO) Part Number Description 4RU 2 x 1.2GHz, 8 Core CPU, 32GB (16 X 2GB), 2x 146GB SAS 2.5”disks, 4x SEVPCJF1Z 1120W PSU, DVD-RW, 2 x Memory Expansion Boards 4 x 1.2GHz, 8 Core CPU, 64GB (32 X 2GB), 2x 146GB SAS 2.5”disks, 4x SEVPGLF1Z 1120W PSU, DVD-RW, 4 x Memory Expansion Boards 4 x 1.4GHz, 8 Core CPU, 128GB (32 X 4GB), 2x 146GB SAS 2.5”disks, SEVPHSF1Z 4x 1120W PSU, DVD-RW, 4 x Memory Expansion Boards ● All configs feature... ● 4 x 1GbE interface, ● 1 x DVD-RW slimline drive ● 8 x PCIe 8 lane slots (2 shared XAUI slots), 2 x16 mechanical ● 4 x USB 2.0 slots (2 front/2 back) ● 1 x DB9 Serial port ● 4 x Redundant, Hot-plug PSUs ● 4 x Redundant, hot-plug fans ● 2 x 146GB 2.5” SAS drives (can accommodate up to 4 drives) ● RAID 0,1 from onboard LSI 1068E controller ● 1 Year On-Site Warranty ● Rack mount kit with cable management arm ● 4 x Power Supply cables (country specific) Sun Internal and Approved Partners Only 20
Sun Fire T5240/T5440 Differences Feature SPARC Enterprise T5240 SPARC Enterprise T5440 Form Factor 2U, 28” deep 4U, 24.5” deep 2x UltraSPARC T2 Plus Up to 4 x UltraSPARC T2 Plus CPU 1.2/1.4 GHz 64-128 threads 1.2/1.4 GHz 128-256 threads FB-DIMM 128GB MAX FB-DIMM, 512GB MAX Memory 32x slots 64x Slots 4x GbE or 4x GbE or Network 3 x GbE + 1 x 10GbE or 3 x GbE + 1 x 10GbE or 2 x GbE + 2 x 10GbE 2 x GbE + 2 x 10GbE Internal Storage Up to 16 x 2.5” SAS, hot-swap Up to 4 x 2.5” SAS, hot-swap Removable Media 1x DVD-RW 1x DVD-RW Serial 1x RS-232, 4x USB 1x RS-232, 4x USB PCI Express slots 6x (low profile) 8x (low profile) 2x 1100W AC/2 x1200Watt DC 4 x 1120W AC Power Supply Redundant/Hot-Swap Redundant/Hot-Swap Fans 5x Redundant/Hot-Swap 4 x Redundant Hot-Swap Sun Internal and Approved Partners Only 21
Front Panel Service Required LED Power/OK LED Locator LED Power Button /Button Front Fan Fault LED System Label PSU Fault LED System Overheat Fault LED Serial Number label DISK 2 DISK 3 DVD-RW DISK 0 DISK 1 Dual USB 2.0 ports Sun Internal and Approved Partners Only 22
Rear Panel PCI Card Securing Latches (internal) PSU Status LED's PSU 3 PSU 2 PSU 1 PCI 0 PCI 1 PCI 2 PCI 3 PCI 4 PCI 5 PCI 6 PCI 7 PCI 0 PCI 2 PCI 3 PCI 5 PCI 6 PCI 7 PSU 0 PCI 0 = PCIe x8, mechanical x8 Quad Gigabit PCI 1 = PCIe x8, mechanical x16 SP Ethernet Ports Serial PCI 2 = PCIe x8, mechanical x8 Port SP NET 2 NET 3 Network PCI 3 = PCIe x8, mechanical x8 POSIX DB-9 Chassis Port Serial Port Status LED's Dual PCI 4 = PCIe x8, mechanical x8, or XAUI 0 Card and Locator Button USB 2.0 NET 0 NET 1 Ports PCI 5 = PCIe x8, mechanical x8, or XAUI 1 Card XAUI 1 XAUI 0 PCI 6 = PCIe x8, mechanical x16 PCI 7 = PCIe x8, mechanical x8 Sun Internal and Approved Partners Only 23
Sun Fire T5440 Block Diagram Mem CPU CPU Mem Mem CPU CPU Mem CPU Module = Module Module Module Module Module Module Module Module - 1 x VF CPU 0 0 2 2 1 1 3 3 - 4 x FB-DIMM Slots Mem Module = - 12 x FB-DIMM Slots Plus Plus Plus Plus T2 T2 T2 T2 x4 VRM VRM VRM VRM VRM VRM VRM VRM SAS LSI1068e x8 SAS 4x FB Links SSI SSI 4x FB Links 4x FB Links SSI SSI 4x FB Links PCIe x8 PCIe x8 PCIe x8 CL0 CL0 PCIe x8 US T2 XBR LPC CL0 0 CL0 x8 CL1 CL1 x8 x8 PLX3 PLX2 x8 US T2 XBR LPC PLX1 PLX0 PEX8548 PEX8548 CL1 CL1 PEX8548 PEX8548 1 x8 x1 x8 CL2 CL2 x8 x8 US T2 XBR Zambezi x8 x8 x8 CL2 2 LPC CL2 x8 x8 PCIe-PCI x8 CL3 CL3 XAUI 1 XAUI 0 US T2 XBR LPC Neptune CL3 3 CL3 USB host x8 QUAD GbE USB Hub USB-IDE 4x SSI PHY LPC FPGA ILOM BCM5466R PCI Card BACK FRONT BACK 0 1 2 3 panel panel panel NET SER DB-9 0 1 2 3 4 USB 2.0 USB 2.0 DVD Sun Internal and Approved Partners Only MGT MGT Quad 10/100/1000 5 6 7 24
Internal View with Cards installed(T5440) System Configuration 4 Fans are required Card (SCC)EEPROM MEM 3 PCIe 7 x8(m)/x8(e) PCIe 6 x16(m)/x8(e) MB/FT0 MB/FT2 MB/FT2 CPU 3 XAUI/PCIe 5 x8(m)/x8(e) CPU 1 MEM 1 Service Processor Daughter Card MEM 2 XAUI/PCIe 4 x8(m)/x8(e) CPU 2 MB/FT3 MB/FT1 MB/FT3 PCIe 3 x8(m)/x8(e) PCIe 2 x8(m)/x8(e) CPU 0 PCIe 1 x16(m)/x8(e) PCIe 0 x8(m)/x8(e) MEM 0 4 x SAS Disk Chassis Power Distribution Power Disk Board Chassis Supplies Power BUS Bars RED = +12Volts BLUE = Ground Chassis Intrusion Switch Extension to top cover from MBD Sun Internal and Approved Partners Only 25
Internal View, cards removed(T5440) System Configuration US T2 Plus XBR Coherency Plane Chips Card (SCC)EEPROM PCIe 7 x8(m)/x8(e) MEM 3 PCIe 6 x16(m)/x8(e) MB/FT0 MB/FT2 MB/FT2 CPU 3 XAUI 1/PCIe 5 x8(m)/x8(e) CPU 1 MEM 1 Service Processor Daughter Card MEM 2 XAUI 0/PCIe 4 x8(m)/x8(e) CPU 2 MB/FT3 MB/FT1 MB/FT3 PCIe 3 x8(m)/x8(e) PCIe 2 x8(m)/x8(e) CPU 0 PCIe 1 x16(m)/x8(e) PCIe 0 x8(m)/x8(e) MEM 0 Power Distribution Power Disk Board Chassis Supplies Chassis SeperatorFence Sun Internal and Approved Partners Only 26
Internal View, Motherboard only(T5440) Neptune Quad DC-DC Converter Modules GbE/10GbE NIC MEM 3 CPU 3 MB/FT0 MB/FT2 MB/FT2 Quad GbE RJ-45 Ports CPU 1 NET MGT port Dual USB 2.0 Port MEM 1 System Status LEDs SER MGT port MEM 2 DB-9 Serial Port CPU 2 Service Processor Card Connector MB/FT3 MB/FT1 CPU 0 MEM 0 PLX PCIe Switch Chips 'Southbridge' Peripheral Chip Fan connectors, with LSI1068E Disk Controller speed control logic Sun Internal and Approved Partners Only 27
Motherboard and CPU/MEM Card layout MEM 3 CPU 3 CPU 1 MEM 1 MEM 2 CPU 2 CPU 0 MEM 0 Motherboard CPU/Mem Board Memory Board Sun Internal and Approved Partners Only 28
Memory Configuration Rules • 1 or 4 CPU cards in a system > Each CPU card has an associated Memory Card Card/Chassis > Unpopulated Cards must be filled Top with spacers > Unpopulated DIMMs must be filled UltraSPARC T2 Plus with spacers VRM CPU • All DIMMS on a CPU & Memory VRM card pair must be the same size BR1/CH0/DO BR1/CH1/DO BANK 0 > 2GB, 4GB or 8GB only (no BR0/CH0/DO VRM Install ALL DIMMs 1GB) BR0/CH1/DO > May support Different DIMM sizes on different pairs... BR1/CH1/D2 BR1/CH1/D3 BANK 3 • 3 Configs... BR1/CH0/D2 BR1/CH0/D3 Install ALL DIMMs > Fill Bank 0 BR0/CH1/D2 BANK 2 > Fill Banks 0,1 VRM BR0/CH1/D3 Install ALL DIMMs BR0/CH0/D2 > Fill Banks 0,1,2,3 BR0/CH0/D3 BR0/CH1/D1 VRM BR0/CH0/D1 BANK 1 • All standard configs provide Install ALL DIMMs BR1/CH1/D1 BR1/CH0/D1 exactly 8 free DIMM slots simultaneously Sun Internal and Approved Partners Only 29
Network Ports and Options XAUI 0 XAUI 1 Device NET 0 NET 1 NET 2 NET 3 XAUI 0 XAUI 1 Driver nxge0 nxge1 nxge2 nxge3 nxge0* nxge1* PCI Device PCI@500/0/c/0,0 PCI@500/0/c/0,1 PCI@500/0/c/0,2 PCI@500/0/c/0,3 PCI@500/0/c/0,0 PCI@500/0/c/0,1 NOTES All Ethernet ports provided by a single Neptune Chip on the motherboard NET 2 NET 3 NET 0 and NET 1 are disabled by XAUI 1 and XAUI0,if installed, to take advantage of 10GbE networking * If a XAUI card is installed, the matching RJ-45 port is disabled (NET 0 and NET 1) XAUI cards take one of three current Fibre 10GbE transceivers Short Reach(SR), Long Reach(LR) and Long Reach Multimode(LRM) NET 0 NET 1 Copper versions of the XAUI card are expected Post-RR XAUI Cards consume the same physical space in the chassis as a PCI card, so are mutually exclusive with PCI cards in the same location Sun Internal and Approved Partners Only 30
10GbE XAUI Option Cards • Interface to the Neptune chip to upgrade two of its ports to 1GbE • Cards each consume a PCI-E slot if installed • Attach to dedicated XAUI sockets on special risers • 10Gbe XAUI Card – SESY7XA1Z > Requires Fibre Tranceivers – 10GbE Short Reach(SR) – SESY7XT1Z – 10GbE Long Reach (LR) – SESY7XT2Z – 10GbE Long Reach multimode (LRM) – part No. TBD • Copper connector XAUI versions to follow post-RR Sun Internal and Approved Partners Only 31
PCIe Slots Built In • No Riser Cards are used, slots are directly on the motherboard • Low Profile PCI-Express cards only • Each Slot is capable of 25Watts power • Total power capacity limited to 140Watts > Systems design point was to accommodate 8 x 17.5Watt Atlas(Quad GbE using Neptune) card • Each Slot has x8 PCIe electrical capability and at least x8 mechanical connectors • Slots 1 and 6 have x16 mechanical connectors • XAUI cards in slots 4 and 5 replace PCIe cards in those slots Sun Internal and Approved Partners Only 32
PCIe External IO Expansion Unit aka Expansion Chassis, I/O Box • Up to 2 x PCI Expansion Chassis are supported > P/N : SENX8BE1Z > 4RU, 19” rack compatible , 28” deep > 1 Host card supported per US T2 Plus > Each chassis is managed by host SP • Only support PCIe card IO boats ( PCI-X ?) > Each 'boat' requires an Link Card in the host (ensign) and a link Card in the boat (captain) with special card between them > IO 'boat' supports 6 x PCIe x8 full-length, hot- pluggable PCIe cards • Chassis default configs contain > 2 x Hot-plug redundant 600Watt PSU/FAN FRU's > 1 x PCIe 'boat', 1 link card pair(host-chassis) and cable > 6 x PCIe card carriers for x8 PCIe Cards > Customers can add a further PCIe 'boat' – P/N : SENX8TE1Z and more Sun Internal and Approved Partners Only 33
Disk Drives • Disk drives are all 2.5” form factor • Original Galaxy NEMO bracket (not Marlin Bracket) • Support SAS only (Can mix SAS/SATA after qual) • SAS Drives – 73GB @ 10K RPM – 146GB @ 10K RPM – 73GB @ 15K RPM – SSD (post-RR) • Same disks as T2x00 systems Disk LED's Ready to Remove Fault Status Sun Internal and Approved Partners Only 34
Disk Controller Options • T5440 has embedded LSI 1068E on the motherboard – JBOD up to 4 disks – Controls internal disks only – Unused 4 ports are not used – RAID 0 (striping) and RAID 1(mirroring) – Hot-plug supported – Hot-swap in non-RAID configs is done with 'cfgadm' • SGXPCIESAS-R-INT-Z NOT supported > Not available for T5440 due to chassis design and only 4 disk capacity • SGXPCIESAS-R-EXT-Z Supported – For external disk arrays only (no internal ports) – 8 port SAS RAID 0,1,1E,5,5EE,6,10,50,60 (depending on number of disks available) – Adaptec Card and Firmware with Intel SunRiseLake chipset – Battery Backed Write Cache (BBWC) of 256MB, at least 72 hours capacity, much longer when new Sun Internal and Approved Partners Only 35
Fans and Cooling • Front to Back cooling • Two rows of fans provide redundancy > PSU fans provide sufficient redundancy for Disk cooling Back Front • Requires 4 Fans Disks PSU's – Main System Cooling MEM0 – MB/FT0 MB/FT2 MB/FT3 MB/FT3 CPU0 – MB/FT1 CPU2 Motherboard MEM2 – MB/FT2 MEM1 MB/FT1 MB/FT0 MB/FT1 – MB/FT3 CPU1 CPU3 – PSU integrated Fan Cooling flow MEM3 – Cools disk drives and PSU's – PSUFAN0 – PSUFAN1 – PSUFAN2 – PSUFAN3 Sun Internal and Approved Partners Only 36
Fans modules and boards Fan Fault Amber LED Lit if fan faulty • 4 Fans Required > 60 Seconds grace if FAN removed for service • Fans are speed controlled by the Service Processor • Speeds are independantly monitored by SP • Speeds tuned to minimise vibrations > Built in vibration sensor • Speeds chosen to minimise harmonic resonance with disk drives and other fans. • Each fan has an Amber FAULT LED – if lit the fan module is faulty Sun Internal and Approved Partners Only 37
Power Supplies • PSU's are high efficiency • Output 12V (and 3.3v standby) PSU Fault • Most DC-DC conversion done in MBD LED • Integrated Fans • 4 PSU's > 2 + 2 Redundancy (Grid Redundancy) > Rated at 1120Watts each. > BEWARE... – Same form factor as SunFire X4600 PSU's ... DO NOT MIX Sun Internal and Approved Partners Only 38
Rack Mounting Options • Two Rack mounting options are available.. • Standard Rack Mount (4 post) > P/N : SEVY9RK1Z > Same as Sun Fire X4600 server > Standard configs contain this kit • Tool-less Rack Kit (4 post) > P/N : SEVY9RK2Z > Optional kit, purchased seperately > Snaps into Sun and most 3rd party Racks > Requires no tools to fit or remove > Will replace tool-required version once available Sun Internal and Approved Partners Only 39
Solaris for Sun SPARC Enterprise T5x40 • T5x40 servers will require Solaris10 5/08 (update 5) with patches > A Patch CD is supplied in the box – Likely to be out of date, so customers should check SunSolve... • This incorporates several specific UltraSPARC T2/T2 Plus optimizations and for CMT servers in general • Also includes some specific T5x40 optimisations > UltraSPARC T2 Plus MPO optimisations > nxge driver tweaks for 10GbE devices > MMU DMA relaxed scheduling > LDOMs pre-install upgrade to version 1.0.3 (US T2 Plus support) Sun Internal and Approved Partners Only 40
Solaris FMA • Now aware of CPU Core offlining • PCIe Fabric diagnostics improved • Crypto Unit offlining • Neptune awareness (though XAUI's not diagnosed) • POST memory CE tolerance improved • FRU identity info improved in fault reports (part/ser #) • Diagnostic Engines (DE) – Existing DE have been extended for T5x40 – New Eversholt DE used for HostBridge, PCIe diagnostics • Limitations... – Can't isolate Uncorrectable Errors to a single DIMM – PCIe riser cards not in topology – XAUI card and XFP not in the topology – SouthBridge chip is not diagnosed at this time (DVD, USB,...) – IO Expansion box is not diagnosed via FMA Sun Internal and Approved Partners Only 41
Agenda • Introduction • Comparision of the UltraSPARC T2 to UltraSPARC VF • Sun SPARC Enterprise T5x40 servers > Overview of Features > System Architecture and Tours > Extra Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 3.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 42
ILOM Features at a Glance • Hardware Control, Inventory, Monitoring • Host Configuration and Administration • SP Configuration and Administration • Logging, Auditing and Alerting • Authentication : – LDAP, RADIUS, Active Directory, Local Users • Services : – SSH, HTTP(S), NTP, Syslog, SMTP • Out-of-Band – Via Serial port, or dedicated Ethernet port • In-Band – Via Host OS, firmware upload, Explorer, Syslog Sun Internal and Approved Partners Only 43
ILOM – Industry Standard Manageability • ILOM enables Sun systems to be easily integrated into third party management infrastructures • Key functionality > IPMI 2.0 > SNMP V1, V2c, V3 > SSH 2.0 > LDAP, RADIUS & MS Active Directory authentication > Browser UI, SMASH DMTF CLP + ALOM style CLI • NOTE : No RKVMS functionality like Galaxy ILOM > No Java plugin required • Browsers tested include, these base versions an later releases – Mozilla 1.7 on Solaris 10, IE 6.0 on W2K Server, Firefox 1.0 and 1.5 on W2K server Sun Internal and Approved Partners Only 44
ILOM – Industry Standard Manageability • Identical to ILOM 2.0 for T5x20 • Default user is 'root' with password 'changeme' > Use for ILOM WEB GUI access and SSH access and IPMI • The ALOM 'admin' user is not defined by default, create this user with an ALOM CLI • Lost 'root' password and ILOM config is reset using jumpers > T5140/T5240 - Motherboard Jumper J10401 (short pins 1-2) > T5440 - SP Daughter card Jumper J1127 (short pins 1-2) • Note... 'sunservice' user shares the password of the 'root' user ONLY FOR SERVICE ACCESS* • Passwords must be 8 to 16 chars (used to be max 8) • Max of 10 users allowed (used to be 16) Sun Internal and Approved Partners Only 45
ILOM Common Tasks • Setting up an ALOM 'admin' user from the ILOM CLI > SUNSP00144F4641BD login: root Password: changeme Waiting for daemons to initialize... Daemons ready Sun(TM) Integrated Lights Out Manager Version 2.0.4.0 Copyright 2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Warning: password is set to factory default. ->create /SP/users/admin role=Administrator cli_mode=alom Creating user... Enter new password: ******** Enter new password again: ******** Created /SP/users/admin2 -> exit SUNSP00144F4641BD login: admin Password: ******** Waiting for daemons to initialize... Daemons ready Sun(TM) Integrated Lights Out Manager Version 2.0.4.0 Copyright 2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. sc> Sun Internal and Approved Partners Only 46
ILOM Common Tasks • Setting up ILOM IP configuration from the ILOM CLI > SUNSP00144F4641BD login: root Password: changeme Waiting for daemons to initialize... Daemons ready Sun(TM) Integrated Lights Out Manager Version 2.0.4.0 Copyright 2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Warning: password is set to factory default. -> set /SP/network state=enabled Set ’state’ to ’enabled’ -> set /SP/network pendingipaddress=10.8.31.163 Set ’pendingipaddress’ to ’10.8.31.163’ -> set /SP/network pendingipdiscovery=static Set ’pendingipdiscovery’ to ’static’ -> set /SP/network pendingipnetmask=255.255.252.0 Set ’pendingipnetmask’ to ’255.255.252.0’ Static IP -> set /SP/network pendingipgateway=10.8.31.248 configuration Set ’pendingipgateway’ to ’10.8.31.248’ -> set /SP/network commitpending=true Set ’commitpending’ to ’true’ -> -> set/SP/pendingipdiscovery=dhcp Set ’pendingipdiscovery’ to ’dhcp’ -> set /SP/network commitpending=true Set ’commitpending’ to ’true’ -> Dynamic IP (DHCP) configuration Sun Internal and Approved Partners Only 47
ILOM IPMI functionality • IPMI is a widely used management protocol for x86 servers • ... But its new for SPARC boxes... • Download the required tools from > The Solaris Supplemental CD (an older binary package) > Latest is available at http://ipmitool.sourceforge.net – Binary packages (usually older versions) are available for Solaris x86 and SPARC, as well as Linux, Windows, Mac...etc > There is a Blueprint available from Sun also (mainly x64 centric, but applies to SPARC now too) – http://www.sun.com/blueprints/0107/820-1011.pdf -bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sdr /MB/V_+1V0_VDD | 1.01 Volts | ok /MB/V_+3V3_STBY | 3.36 Volts | ok Get the Sensor /MB/V_+3V3_MAIN | 3.36 Volts | ok Data /MB/V_+5V0_VCC | 4.97 Volts | ok Repository /MB/V_+12V0_MAIN | 12.16 Volts | ok .. .. -bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sunoem sbled set /SYS/LOCATE on /SYS/LOCATE | ON Set the System -bash-ventus2$ Locator LED flashing Sun Internal and Approved Partners Only 48
ILOM Common Tasks • Logging into the ILOM Web GUI • Point Web Browser at the IP address of the ILOM > Will automatically redirect to a secure HTTPS port > Best to disable any Web Proxies for best performance > Make sure port 80, 443 are open on any firewall • Log in as any defined user, default Administration user is 'root' Sun Internal and Approved Partners Only 49
ILOM/ALOM Common Tasks • A Tour of the ILOM Web GUI • GUI provides ways to set up IP address configurations for ILOM • Configure additional ILOM/ALOM users • Access Sensor Readings • Control System configurations • Carry out firmware updates. • etc.... Sun Internal and Approved Partners Only 50
ALOM 2.0 vs. 1.4 CLI Differences • Default CLI appearance is ILOM – User is 'root' – Must create an 'admin' Administrator user with the ALOM CLI • Flashupdate in new ALOM expects a TFTP server not an FTP server • ALOM IP address changes now active only after running ... – 'setsc netsc_commit true' – Previously changes were active only after SC reset – DHCP is enabled by default • View firmware versions with – 'showhost version' sc> showhost version Sun System Firmware 7.0.0_build_13 2007/08/10 23:44 • Only Admin and Operator roles Host flash versions: Hypervisor 1.5.0.build_13**PROTOTYPE** 2007/08/10 22:54 OBP 4.27.0.build_04***PROTOTYPE BUILD*** 2007/08/10 13:25 are defined Sun Fire[TM] Huron POST 4.27.0.build_04***PROTOTYPE BUILD*** 2007/08/10 13:51 Sun Internal and Approved Partners Only 51
ALOM backward Compatibilty CLI • Uses ALOM-CMT 1.2 for a base feature set • Includes ALOM-CMT 1.4 features... > break -d > Host Watchdog Timer > Bootmode LDOMs awareness > Forward SP events to Hot/Solaris syslog via inband channel Sun Internal and Approved Partners Only 52
Agenda • Introduction • Comparision of the UltraSPARC T2 to UltraSPARC VF • Sun SPARC Enterprise T5x40 servers > Overview of Features > Comparison to SunFire T1000/T2000/T5x20 > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 2.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 53
Logical Domains 1.0.3 Features • UltraSPARC T2 Plus Platform Support > 128 domains supported at RR – 256 domains work on T5440, may be qualified later, requires Solaris fixes • Feature Set very similar to 1.0.2 • I/O domain can reboot while the other domains keep running > Dependent domains would wait for I/O transactions to complete • LDoms SNMP MIB > Modeled after industry standard DMTF CIM (Common Information Model) > Running on control domain to enable remote SNMP monitoring • Domain minimization support > Minimal/reduced Solaris installation > Base minimal packages plus optional ones for guest, control, I/O or service domains Sun Internal and Approved Partners Only 54
Logical Domains 1.0.3 Requirements • Solaris 10 05/08 (update 5) and required patches • Systems firmware version – T5x40 Firmware 7.1.5b = SP FW 2.0.4.20+ – T5x20/SB6320 Firmware 7.0.4+ = SP FW 2.0.4.0+ – T2000/T1000/T6300/Pelton Firmware 6.5.x • SUNWldm.v LDOMs Manager Package – Version 1.0.3 Sun Internal and Approved Partners Only 55
Agenda • Introduction • Comparision of the UltraSPARC T2 to UltraSPARC VF • Sun SPARC Enterprise T5x40 servers > Overview of Features > Comparison to SunFire T1000/T2000/T5x20 > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 2.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 56
Option Parts List for T5440(ATO) Part Number Option Description Available SEVY1CB1Z T2 Plus CPU module, 8 core, 1.2GHz, 8GB(4x2GB) with memory board RR SEVY1CC1Z T2 Plus CPU module, 8 core, 1.2GHz, 16GB(4x4GB) with memory board RR SEVY1CD1Z T2 Plus CPU module, 8 core, 1.2GHz, 32GB(4x8GB) with memory board SEVY1DD1Z T2 Plus CPU module, 8 core, 1.4GHz, 32GB(4x8GB) with memory board SEVY1DB1Z T2 Plus CPU module, 8 core, 1.4GHz, 8GB(4x2GB) with memory board Subject to 1.4GHz availability SEVY1DC1Z T2 Plus CPU module, 8 core, 1.4GHz, 16GB(4x4GB) with memory board Subject to 1.4GHz availability SESY2A1Z 2GB Memory Expansion kit(2x1GB) SESY2B2Z 4GB Memory Expansion kit(2x2GB) Non-Extended ECC version, no mixing SESY2D1Z 16GB Memory Expansion kit (2 X8 GB) SESY2C1Z 8GB Memory Expansion kit(2x4GB) SESVY3A11Z 73GB 10Krpm, 2.5” SAS Drive SESVY3C11Z 146GB 10Krpm, 2.5” SAS Drive SESVY3A21Z 73GB 15Krpm, 2.5” SAS Drive SEVY9PS51Z 1120W AC Power Supply SEVY9DV1Z DVD, 8x DVD-RW Slimline module SEVY9RK1Z Slide Rail Kit, Screw Mounting SEVY9RK2Z Slide Rail Kit, Toolless Mounting Sun Internal and Approved Partners Only 57
Option Parts List for T5440 (cont..) Part Number Option Description Avialable Max number SG-XPCIE1FC-QF4 Qlogic, 4Gb, PCI-E Single Port RR 8 SG-XPCIE2FC-QF4 Qlogic, 4Gb, PCI-E Dual Port RR 8 SG-XPCIE1FC-EM4 Emulex, 4Gb, PCI-E Single Port RR 8 SG-XPCIE2FC-EM4 Emulex, 4Gb, PCI-E Dual Port RR 8 SG-XPCIE2SCSI-U320Z U320 SCSI, PCI-E Dual Port RR 8 SG-XPCIE8SAS-E-Z SAS, 8 Port, PCI-E RR 8 SGXCPCIESAS-R-EXT-Z SAS, 8 Port External RAID Controller RR 2 X7281A-2 Northstar Dual port GigE MMF Low profile PCI-E RR 8 X7280A-2 Northstar Dual port GigE UTP Low profile PCI-E RR 8 X1236A-Z Infiniband HBA, Dual 4x ports, PCI-E RR 8 X4447A-Z Atlas Quad Port, GigE Copper, PCI-E (up to 3/6) RR 8 X1027A-Z Atlas Dual Port, 10GigE, PCI-E RR 4 X5558A Tranceiver 10GE SR XFP (XAUI/Atlas) RR - X5560A-Z Tranceiver 10GE LR XFP (XAUI/Atlas) RR - X3000A-Z XVR-300, PCI-E x8 Graphics cards (up to 1) RR 4 SESY7XA1Z 10GbE XAUI card RR 2 TBD 10GbE XAUI Copper 10g Base-T TBD 2 X6000A Sun Crypto Accelerator 6000 RR 2 SENY8BE1Z External IO Expansion Unit – PCI-E base Config RR 2 SENY8TE1Z PCI-Express “boat” for External IO Expansion Unit (6 Hot Plug carriers) RR 4 SELY8LK1Z Copper link kit for IO Expansion unit “boat” RR 4 Sun Internal and Approved Partners Only 58
Agenda • Introduction • Comparision of the UltraSPARC T2 to UltraSPARC VF • Sun SPARC Enterprise T5x40 servers > Overview of Features > Comparison to SunFire T1000/T2000/T5x20 > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 2.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 59
Tuning Out of the Box • Using Solaris 10 05/08 (update 5) with Patches > MPO optimisations used > DMA relaxed Ordering implemented > Default pages sizes for – Initdata Segments is 64kB pages – Private Anon, Heap, Process Stack, Text, Shared Anon segments are 4MB pages – ISM and DISM use 256MB pages – New variables introduced to control these sizes if changes are needed – max_uheap_lpsize,max_ustack_lpsize, max_privmap_lpsize, max_shm_lpsize, max_uidata_lpsize, max_utext_lpsize, max_shm_lpsize > L2 Cache hashing implemented – helps randomise accesses to L2 caches and ease thrashing > Stack biasing implemented – D-Cache is only 4 Way associative... 8 threads – Running multiple copies of a binary can lead process stacks aligning in the cache and to hot lines in the cache – Stack Bias slews the stack variable for processes. Sun Internal and Approved Partners Only 60
Tuning Out of the Box (continued..) > Processor Groups introduced – Helps associate CPUs with some shared resources (int/FP pipelines, Crypto units etc) – Enables improved multi-level CMT load balancing and affinity policies > 'bcopy'/'uiomove' Optimizations – Key kernel operations – new implementation about 2x faster than previous • Optimisations will require patching of S10 05/08 > These will be post RR patches as they become available > Most fixes are already committed in update 5 > Check patch levels for customers with performance issues > Also check Firmware versions as this can improve performance too... Sun Internal and Approved Partners Only 61
Tuning Recommendations • Less involved than with T2000 • S10 5/08 is more 'automatic' – OOB optimised page size selection, thread scheduling, block copy, IP instances for zones, page coloring etc... • If tuning is necessary remember... > Improved pipeline results in Single threaded performance around 20% better > Floating Point is no longer an issue > Start with T2000 tuning options > Use updated trapstat, cpustat, • LDOMs may be an easier way to scale poorly threaded applications • Solaris Crypto framework is still being tuned > Great for big packets, working on improving small packets • A Blueprint is being prepared (Denis Sheahan) Sun Internal and Approved Partners Only 62
Tuning Recommendations (cont..) • Refer to UltraSPARC T1 applications tuning best practices... > http://www.sun.com/servers/coolthreads/tnb/applications.jsp • CoolTools are being updated for UltraSPARC T2 and T2 Plus > http://www.sun.com/servers/coolthreads/overview/cooltools.jsp • Sun Studio 12 is UltraSPARC T1 and T2/T2 Plus aware > Recompiling without code changes should benefit T1/T2/T2 Plus apps • LDOMs configs can benefit significantly from a transition to T5x20 platform > Multiple IO/Service domains with higher bandwidth PCI busses > Make sure appropriate LDOMs tuning and sizing is carried out • Wider Crypto capability, so ensure code is using PKCS#11 framework > Ensure that the ncp (SSL, RSA, DSA etc) and n2cp (bulk ciphers), n2rng(random number generator) drivers are used appropriately Sun Internal and Approved Partners Only 63
PCIe Performance • Easily two to four times the performance of T5x20 (two or four PCIe Busses) • All PCI card slots are x8 PCI electrically > Single level hierarchy > Cut through switching > PCI-E relaxed ordering for higher performance • Performance per bus... > Marketing performance = 4000MB/sec > Theoretical Rate with overheads = 3790MB/sec > Max. DMA Achievable(bidirectional) = 3096MB/sec – with 128byte packets – And 512 byte read size Sun Internal and Approved Partners Only 64
Agenda • Introduction • Comparision of the UltraSPARC T2 to UltraSPARC VF • Sun SPARC Enterprise T5x40 servers > Overview of Features > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 2.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 65
Tools and Information • Cool Tools – http://www.sun.com/servers/coolthreads/overview/cooltools.jsp • SUNWIN Tokens > Sun SPARC Enterprise T5440 – Architecture Whitepaper : 526122 – JTF : 526118 > T5440 RAS Whitepaper : TBA • Video Tours > Check the Sun.com product pages • Power Calculators > Check the Sun.com product pages Sun Internal and Approved Partners Only 66
Further Information • Presto > T5440 > http://sunwebcollab.east.sun.com/gm/document-1.26.680780/Batoka_PRESTO.html • Installed Business Base (IBB) Promotions > http://sunwebcollab.east.sun.com/gm/document-1.26.680782/Batoka_IBB_Promo.html • Start Up Essentials program > http://sunwebcollab.east.sun.com/gm/document-1.26.680794/Maramba_Batoka_SU_April1.html Sun Internal and Approved Partners Only 67
Reference Information • T5440 Engineering TOI – http://ittdev.east.sun.com/TechTalk2/EngineeringTOIs/ • ILOM/ALOM User Guides – http://www.sun.com/products-n-solutions/hardware/docs/pdf/819-1160-13.pdf – http://www.sun.com/products-n-solutions/hardware/docs/pdf/819-7981-10.pdf • Solaris FMA – http://fma.eng – http://lightside.sfbay/n2-fma/docs.php • Sun Internal and Approved Partners Only 68
Thank You Sun Microsystems 69
T5x40 PCI Topology Sun Internal and Approved Partners Only 70
10GbE Networking Substantiation 10GbE price for 2 x SR-fiber PCI-E adaptors = $6,740. From HP DL380G5 pricing on HP Store, www.hp.com, 12/17/07. Cost for 2 x 10GbE and SR transceivers for Maramba and Batoka are $1,800. From sun.com, 12/17/07 Sun Internal and Approved Partners Only 71

More Related Content

PPT
Design of small to large multi site solution, based on 8300, 8800 simplex, 88...
byAURACA
 
PDF
Track SD-WAN.pdf
byMarceloTuxSP
 
PDF
FIDOセキュリティ認定の概要と最新状況
byFIDO Alliance
 
PDF
Roadmap to CompTIA Security+
byMansi Kandari
 
PDF
Why did the shift-left end up in the cloud for Bank Julius Baer?
byBATbern
 
PDF
Soal Linux Actual - ITNSA LKS SMK Tingkat Provinsi NTB 2021
byI Putu Hariyadi
 
PDF
Konfigurasi Site-to-Site IPSec VPN Tunnel di Mikrotik menggunakan GNS3
byI Putu Hariyadi
 
PPTX
Konsep dasar management bandwidth-AIJTKJGNC
byAsepSukarya2
 
Design of small to large multi site solution, based on 8300, 8800 simplex, 88...
byAURACA
 
Track SD-WAN.pdf
byMarceloTuxSP
 
FIDOセキュリティ認定の概要と最新状況
byFIDO Alliance
 
Roadmap to CompTIA Security+
byMansi Kandari
 
Why did the shift-left end up in the cloud for Bank Julius Baer?
byBATbern
 
Soal Linux Actual - ITNSA LKS SMK Tingkat Provinsi NTB 2021
byI Putu Hariyadi
 
Konfigurasi Site-to-Site IPSec VPN Tunnel di Mikrotik menggunakan GNS3
byI Putu Hariyadi
 
Konsep dasar management bandwidth-AIJTKJGNC
byAsepSukarya2
 

Viewers also liked

PDF
Sun sparc enterprise t5140 and t5240 servers technical presentation
byxKinAnx
 
PDF
Sun fire x4140, x4240, x4440 server customer presentation
byxKinAnx
 
PDF
Sun sparc enterprise t5120 and t5220 servers technical presentation
byxKinAnx
 
PPTX
Power Searching Google And Google Scholar
byLaura Wilson
 
PPTX
Routing over ericsson mini link
byAhmed Nabeeh
 
PDF
Game Based Learning for Language Learners
byShelly Sanchez Terrell
 
PPTX
Connecting With the Disconnected
byChris Wejr
 
Sun sparc enterprise t5140 and t5240 servers technical presentation
byxKinAnx
 
Sun fire x4140, x4240, x4440 server customer presentation
byxKinAnx
 
Sun sparc enterprise t5120 and t5220 servers technical presentation
byxKinAnx
 
Power Searching Google And Google Scholar
byLaura Wilson
 
Routing over ericsson mini link
byAhmed Nabeeh
 
Game Based Learning for Language Learners
byShelly Sanchez Terrell
 
Connecting With the Disconnected
byChris Wejr
 

Similar to Sun sparc enterprise t5440 server technical presentation

PDF
Amd accelerated computing -ufrj
byRoberto Brandao
 
PPTX
Trend - HPC-29mai2012
byAgora Group
 
PDF
Maximizing Application Performance on Cray XT6 and XE6 Supercomputers DOD-MOD...
byJeff Larkin
 
PDF
HPCMPUG2011 cray tutorial
byJeff Larkin
 
PPTX
Intel_Embedded Intel Core Processors Do More Now and in the Future
byIşınsu Akçetin
 
PPTX
8-way-server
byMarco van der Hart
 
PDF
Sun fire x2250 technical training presentation
byxKinAnx
 
PDF
GPUDirect RDMA and Green Multi-GPU Architectures
byinside-BigData.com
 
PDF
GA1000- Entry level gaming computer
byAEWIN
 
PPTX
Intel_Low Power Intelligent Solutions with Intel Atom Processor
byIşınsu Akçetin
 
PDF
GA 5000- High Computing Gambling PC for mulit-player
byAEWIN
 
PDF
Sun Microsystems
byguest09c59b06
 
PDF
Sandy bridge platform from ttec
byTTEC
 
PDF
20130218 company profile and q1 product roadmap
bySirena Cheng
 
PPT
System x M4
byNatalija Pavic
 
PPTX
Bobcat to jaguar_v2
byAMDPhil
 
PDF
Nucleus RM Rear IO
byjledwell
 
PDF
Ibm power7
byTom Presotto
 
PPTX
HP - HPC-29mai2012
byAgora Group
 
PDF
Nucleus RM Front IO
byjledwell
 
Amd accelerated computing -ufrj
byRoberto Brandao
 
Trend - HPC-29mai2012
byAgora Group
 
Maximizing Application Performance on Cray XT6 and XE6 Supercomputers DOD-MOD...
byJeff Larkin
 
HPCMPUG2011 cray tutorial
byJeff Larkin
 
Intel_Embedded Intel Core Processors Do More Now and in the Future
byIşınsu Akçetin
 
8-way-server
byMarco van der Hart
 
Sun fire x2250 technical training presentation
byxKinAnx
 
GPUDirect RDMA and Green Multi-GPU Architectures
byinside-BigData.com
 
GA1000- Entry level gaming computer
byAEWIN
 
Intel_Low Power Intelligent Solutions with Intel Atom Processor
byIşınsu Akçetin
 
GA 5000- High Computing Gambling PC for mulit-player
byAEWIN
 
Sun Microsystems
byguest09c59b06
 
Sandy bridge platform from ttec
byTTEC
 
20130218 company profile and q1 product roadmap
bySirena Cheng
 
System x M4
byNatalija Pavic
 
Bobcat to jaguar_v2
byAMDPhil
 
Nucleus RM Rear IO
byjledwell
 
Ibm power7
byTom Presotto
 
HP - HPC-29mai2012
byAgora Group
 
Nucleus RM Front IO
byjledwell
 

More from xKinAnx

PPTX
Engage for success ibm spectrum accelerate 2
byxKinAnx
 
PPTX
Accelerate with ibm storage ibm spectrum virtualize hyper swap deep dive
byxKinAnx
 
PDF
Software defined storage provisioning using ibm smart cloud
byxKinAnx
 
PDF
Ibm spectrum virtualize 101
byxKinAnx
 
PDF
Accelerate with ibm storage ibm spectrum virtualize hyper swap deep dive dee...
byxKinAnx
 
PDF
04 empalis -ibm_spectrum_protect_-_strategy_and_directions
byxKinAnx
 
PPTX
Ibm spectrum scale fundamentals workshop for americas part 1 components archi...
byxKinAnx
 
PPTX
Ibm spectrum scale fundamentals workshop for americas part 2 IBM Spectrum Sca...
byxKinAnx
 
PPTX
Ibm spectrum scale fundamentals workshop for americas part 3 Information Life...
byxKinAnx
 
PPTX
Ibm spectrum scale fundamentals workshop for americas part 4 Replication, Str...
byxKinAnx
 
PPTX
Ibm spectrum scale fundamentals workshop for americas part 4 spectrum scale_r...
byxKinAnx
 
PPTX
Ibm spectrum scale fundamentals workshop for americas part 5 spectrum scale_c...
byxKinAnx
 
PPTX
Ibm spectrum scale fundamentals workshop for americas part 6 spectrumscale el...
byxKinAnx
 
PPTX
Ibm spectrum scale fundamentals workshop for americas part 7 spectrumscale el...
byxKinAnx
 
PPT
Ibm spectrum scale fundamentals workshop for americas part 8 spectrumscale ba...
byxKinAnx
 
PPTX
Ibm spectrum scale fundamentals workshop for americas part 5 ess gnr-usecases...
byxKinAnx
 
PDF
Presentation disaster recovery in virtualization and cloud
byxKinAnx
 
PDF
Presentation disaster recovery for oracle fusion middleware with the zfs st...
byxKinAnx
 
PDF
Presentation differentiated virtualization for enterprise clouds, large and...
byxKinAnx
 
PDF
Presentation desktops for the cloud the view rollout
byxKinAnx
 
Engage for success ibm spectrum accelerate 2
byxKinAnx
 
Accelerate with ibm storage ibm spectrum virtualize hyper swap deep dive
byxKinAnx
 
Software defined storage provisioning using ibm smart cloud
byxKinAnx
 
Ibm spectrum virtualize 101
byxKinAnx
 
Accelerate with ibm storage ibm spectrum virtualize hyper swap deep dive dee...
byxKinAnx
 
04 empalis -ibm_spectrum_protect_-_strategy_and_directions
byxKinAnx
 
Ibm spectrum scale fundamentals workshop for americas part 1 components archi...
byxKinAnx
 
Ibm spectrum scale fundamentals workshop for americas part 2 IBM Spectrum Sca...
byxKinAnx
 
Ibm spectrum scale fundamentals workshop for americas part 3 Information Life...
byxKinAnx
 
Ibm spectrum scale fundamentals workshop for americas part 4 Replication, Str...
byxKinAnx
 
Ibm spectrum scale fundamentals workshop for americas part 4 spectrum scale_r...
byxKinAnx
 
Ibm spectrum scale fundamentals workshop for americas part 5 spectrum scale_c...
byxKinAnx
 
Ibm spectrum scale fundamentals workshop for americas part 6 spectrumscale el...
byxKinAnx
 
Ibm spectrum scale fundamentals workshop for americas part 7 spectrumscale el...
byxKinAnx
 
Ibm spectrum scale fundamentals workshop for americas part 8 spectrumscale ba...
byxKinAnx
 
Ibm spectrum scale fundamentals workshop for americas part 5 ess gnr-usecases...
byxKinAnx
 
Presentation disaster recovery in virtualization and cloud
byxKinAnx
 
Presentation disaster recovery for oracle fusion middleware with the zfs st...
byxKinAnx
 
Presentation differentiated virtualization for enterprise clouds, large and...
byxKinAnx
 
Presentation desktops for the cloud the view rollout
byxKinAnx
 

Recently uploaded

PDF
Generative AI in 2026: Hype, Bubble, Winter or The Real Deal?
byDr. Tathagat Varma
 
PDF
The major tech developments for 2026 by Pluralsight, a research and training ...
byChris Skinner
 
PDF
MuleSoft Vibes is an AI-powered assistant
byVikalp Bhalia
 
PDF
The year in review - MarvelClient in 2025
bypanagenda
 
PDF
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
PDF
DIGITAL FORENSICS - Notes for Everything.pdf
bypankajkumavatbeit
 
DOCX
Cloud Security, Serverless Security. Cybersecurity
byEdcelPacayraDuena
 
PPTX
The Landscape of Computer Software Development Companies
byYash Singh
 
PDF
How Mobile Apps Are Shaping the Future of Digital Innovation
byWilliam Taylor
 
PPTX
Ethics in AI - Artificial Intelligence Fundamentals.pptx
byemenyiblessing
 
PPTX
From Backup to Resilience: How MSPs Are Preparing for 2026
byMSP360
 
PDF
Empowering Productivity with Clever Prompts and Intelligent Agents
byUni Systems S.M.S.A.
 
PDF
Day 1 - Cloud Security Strategy and Planning ~ 2nd Sight Lab ~ Cloud Security...
by2nd Sight Lab
 
PDF
ElyriaSoftware — Powering the Future with Blockchain Innovation
byElyria Software
 
PDF
Recursive Self Improvement vs Continuous Learning
byBob Marcus
 
PDF
TrustArc Webinar - Looking Ahead: The 2026 Privacy Landscape
byTrustArc
 
PPTX
Ritesh_kumar_Aatmanirbhar Bharat: Make in India, Make for the World.pptx
byriteshrkgs2008
 
PDF
Dev Dives: AI that builds with you - UiPath Autopilot for effortless RPA & AP...
byUiPathCommunity
 
PPTX
Unit-4-ARTIFICIAL NEURAL NETWORKS.pptx ANN ppt Artificial neural network
byssuserd4481a
 
DOCX
Introduction to the World of Computers (Hardware & Software)
byALIABBAS ABASOV
 
Generative AI in 2026: Hype, Bubble, Winter or The Real Deal?
byDr. Tathagat Varma
 
The major tech developments for 2026 by Pluralsight, a research and training ...
byChris Skinner
 
MuleSoft Vibes is an AI-powered assistant
byVikalp Bhalia
 
The year in review - MarvelClient in 2025
bypanagenda
 
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
DIGITAL FORENSICS - Notes for Everything.pdf
bypankajkumavatbeit
 
Cloud Security, Serverless Security. Cybersecurity
byEdcelPacayraDuena
 
The Landscape of Computer Software Development Companies
byYash Singh
 
How Mobile Apps Are Shaping the Future of Digital Innovation
byWilliam Taylor
 
Ethics in AI - Artificial Intelligence Fundamentals.pptx
byemenyiblessing
 
From Backup to Resilience: How MSPs Are Preparing for 2026
byMSP360
 
Empowering Productivity with Clever Prompts and Intelligent Agents
byUni Systems S.M.S.A.
 
Day 1 - Cloud Security Strategy and Planning ~ 2nd Sight Lab ~ Cloud Security...
by2nd Sight Lab
 
ElyriaSoftware — Powering the Future with Blockchain Innovation
byElyria Software
 
Recursive Self Improvement vs Continuous Learning
byBob Marcus
 
TrustArc Webinar - Looking Ahead: The 2026 Privacy Landscape
byTrustArc
 
Ritesh_kumar_Aatmanirbhar Bharat: Make in India, Make for the World.pptx
byriteshrkgs2008
 
Dev Dives: AI that builds with you - UiPath Autopilot for effortless RPA & AP...
byUiPathCommunity
 
Unit-4-ARTIFICIAL NEURAL NETWORKS.pptx ANN ppt Artificial neural network
byssuserd4481a
 
Introduction to the World of Computers (Hardware & Software)
byALIABBAS ABASOV
 

Sun sparc enterprise t5440 server technical presentation

  • 1.
    Sun SPARC Enterprise T5440 RackServers Train-the-Trainer Course Presenter Sun Microsystems, Inc. Sun Confidential CDA Required 1
  • 2.
    NDA Notification This presentationcontains information related to projects that are currently in planning and/or development stages. This information represents the current intentions of Sun Microsystems, Inc. However, all aspects of these projects, including, but not limited to funding, availability, shipping dates, configurations, capacities, performance, and all other characteristics are subject to change and/or cancellation without notice. This material is confidential to Sun Microsystems and should be disclosed to non- employees only under the terms of an executed confidential-disclosure agreement. Distribution of this material to Sun Field Representatives should be done through the GSO CDA process. Under no circumstances should these roadmaps be given directly to a Sun Field Representative. The information contained herein is considered privileged, confidential and protected from disclosure. If you are not the intended recipient or their employee/agent responsible for its delivery, you are requested to refrain from any duplication or distribution of this message or its contents, in respect of the rights of the designated recipient and sender and to delete it from your computer. Sun Internal and Approved Partners Only 2
  • 3.
    Agenda • Introduction • Comparisionof the UltraSPARC T2 to UltraSPARC T2 Plus • Sun SPARC Enterprise T5440 server > Overview of Features > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 2.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 3
  • 4.
    Industry's Most HighlyThreaded Servers Maximum Threading = Higher Throughput, Greater Energy & Space Efficiency T5440 Up to 256 Threads T5140/T5240 Up to 128 Threads T5120/T5220 Up to 64 Threads SB6320 US T2 blade Sun Internal and Approved Partners Only 4
  • 5.
    UltraSPARC T2: • Up to 8 cores @1.2GHz or 1.4GHz TrueSystem On a Chip • Up to 64 threads per CPU • Up to 16 FB-DIMMs, 4 memory controllers FB DIMM FB DIMM FB DIMM FB DIMM > Up to 64GB memory (4GB DIMMs) > 2.5x memory BW of UltraSPARC T1 FB DIMM FB DIMM FB DIMM FB DIMM • 8 x fully pipelined Floating Point units / core, 1 per core MCU MCU MCU MCU • Dual 10Gbit Ethernet and PCI-E integrated onto chip L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$ • 4MB L2 (8 banks) 16 way associative Full Cross Bar • Enhanced MAU/Security co-processor C0 C1 FPU FPU C2 FPU C3 FPU C4 FPU C5 FPU C6 FPU C7 FPU per core MAU MAU MAU MAU MAU MAU MAU MAU > DES, 3DES, AES, RC4, SHA1, DMA Sub-System SHA256, MD5, RSA to 2048 key, ECC,CRC32 NIU Sys I/F • Advanced Power saving features (E-net+) Buffer Switch Core PCI-E • 65nm process technology Power ~95W <1.5W/thread x8 @2.5GHz 2x 10GE Ethernet 4GBytes/s bi-directional Sun Internal and Approved Partners Only 5
  • 6.
    UltraSPARC T2 Processor SPC SPARCv9 CMT Core L2 Data L2 Data Bank 0 Bank 4 CCX Core-Cache Crossbar L2B0 SPARC SPARC SPARC SPARC L2B4 L2$ Level 2 Cache Bank L2 Data Core 0 Core 1 Core 5 Core 4 L2 Data CLC Coherence & Link Controller Bank 1 Bank 5 MCU Memory Control Unit L2B1 L2B5 SIU System Interface Unit L2 L2 L2 L2 MCU0 MCU2 NCU Non-cacheable Unit NCU TAG0 TAG1 TAG5 TAG4 FSR MAC Media Access Controller EFU MCU1 MCU3 DMU Data Management Unit SIO SII L2B3 CCX L2B7 FSR CCU PEU PCI Express Unit L2 Data L2 Data Bank 3 Bank 7 L2 L2 L2 L2 L2B2 L2B6 TAG2 TAG3 TAG7 TAG6 L2 Data L2 Data Bank 2 Bank 6 SPARC SPARC SPARC SPARC DMU Core 2 Core 3 Core 7 Core 6 RDP TDS PEU RTX PSR ESR FSR MAC For a video tour with Rick Hetherington, go to http://www.youtube.com/watch?v=B8DNexYcsa4 Sun Internal and Approved Partners Only 6
  • 7.
    UltraSPARC T2 Plus: • Per Socket Multi-Socket US T2 > Up to 8 cores @1.2GHz or Each Coherency Link 6.4GBytes/s per direction, 12.8GB/s total 1.4GHz 4 x Coherency links give Snoop bandwidth of 51.2GB/s > Up to 64 Threads per socket 4x Coherency > 4MB L2$ 8 banks x 16 Way SA Links > Up to 8 FPU's, 1 per core FB-DIMM FB-DIMM FB-DIMM FB-DIMM > Up to 8 Crypto cores, 1 per core FB-DIMM FB-DIMM FB-DIMM FB-DIMM > DES, 3DES, AES, RC4, SHA1, SHA256, MD5, RSA to 2048 key, ECC,CRC32 MCU MCU > 2 memory Controllers, 16 DIMMS CU CU CU CU > 25Read+13WriteGB/s Memory BW L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$ > One x8 PCIe interface Full Cross Bar • Per system CO C1 C2 C3 C4 C5 C6 C7 > 2 sockets glue-less FPU MAU FPU MAU FPU MAU FPU MAU FPU MAU FPU MAU FPU MAU FPU MAU > 4 sockets using UST2 Plus XBR > 16 or 32 PCIe lanes on 2 or 4 System I/F PCI-Express Socket systems • 65nm process technology SSI x8 @2.5GHz Bus Power ~105W 4GBytes/s <1.5W/thread bi-directional Sun Internal and Approved Partners Only 7
  • 8.
    UltraSPARC T2 PlusProcessor SPC SPARCv9 CMT Core CCX Core-Cache Crossbar L2$ Level 2 Cache Bank CLC Coherence & Link Controller MCU Memory Control Unit SIU System Interface Unit NCU Non-cacheable Unit NCX Non-cacheable Crossbar DMU Data Management Unit PEU PCI Express Unit Sun Internal and Approved Partners Only 8
  • 9.
    CMT Processor Comparison UltraSPARC T1 UltraSPARC T2 UltraSPARC T2 Plus Technology 90nm 65nm 65nm # of Cores 8 8 8 # of Threads 32 64 64 # of Sockets 1 1 1–4 Core Frequency (Ghz) 1 – 1.4 1.2 – 1.4 1.2 – 1.4 Execution pipelines per core 1 2 2 Instructions per cycle per pipe 1 1 1 FPUs per chip 1 8 8 I$ size (per core) 16KB 16KB 16KB D$ size (per core) 8KB 8KB 8KB L2$ size/Set Association 3MB/12-Way 4MB/16-Way 4MB/16-Way Memory Type DDR2 FBDIMM-1 FBDIMM-1 Memory Bandwidth (GB/s) 21.3 42READ + 21WRITE 25READ + 13WRITE Coherence Bandwdth (GB/s) - - 4 x (6.4READ +6.4WRITE) IO Bus JBUS x8 PCIe x8 PCIe IO Bandwidth (GB/s) 2.5 2+2 2+2 On-chip Networking - 2 x 10GE - Crypto Acceleration RSA RSA, RC4, DES, 3DES, AES, SHA1-256, MD5 Power (Watts, Watts/thread) 72, 2.25 95, 1.5 105, 1.6 * Sun Internal and Approved Partners Only 9
  • 10.
    UltraSPARC T2 Plus2-Socket System Coherency Links Dual Channel Dual Channel Dual Channel Dual Channel FBDIMM FBDIMM FBDIMM FBDIMM Memory Controller Memory Controller Memory Controller Memory Controller Coherence Coherence Coherence Coherence Coherence Coherence Coherence Coherence Unit Unit Unit Unit Unit Unit Unit Unit UltraSPARC T2 Plus Cores, Crossbar, L2$ UltraSPARC T2 Plus Cores, Crossbar, L2$ (8 cores, 64 threads, 4MB L2$) (8 cores, 64 threads, 4MB L2$) PCI-Express NCU, DMU NCX PCI-Express NCU, DMU NCX System IO (Network, Disk, etc.) Sun Internal and Approved Partners Only 10
  • 11.
    UltraSPARC T2 Plus4-Socket System 6.4GB/s 6.4GB/s Dual Channel FBDIMM Dual Channel US T2 Plus US T2 Plus FBDIMM PCI-Express PCI-Express 8 Cores 6.4GB/s 6.4GB/s 8 Cores 6.4GB/s 6.4GB/s 64 Threads Coherence 64 Threads Dual Channel 4MB L2$ 4MB L2$ Coherence(4) Hubs FBDIMM Dual Channel FBDIMM Coherence Coherence UltraSPARC Hubs (4) Hubs (4) HubsPlus T2 (4) 6.4GB/s XBR 6.4GB/s 44 ports/hub ports/hub 6.4GB/s 4 ports/hub 6.4GB/s Dual Channel inin each dir each dir FBDIMM Dual Channel in each dir FBDIMM PCI-Express US T2 Plus VF 4 ports/hub US T2 Plus PCI-Express 6.4GB/s per port 6.4GB/s 6.4GB/s 8 Cores 8 Cores in each dir 6.4GB/s 8 Cores 6.4GB/s 64 Threads 8 Threads/Core 64 Threads Dual Channel FBDIMM 4MB L2$ 4MB L2$ Dual Channel 4MB L2$ FBDIMM 6.4GB/s 6.4GB/s Sun Internal and Approved Partners Only 11
  • 12.
    US T2 PlusXBR Coherency Hub Chips... • 4 XBR chips in a system • XBR routes non-cacheable PIOs to the • Each operates in its own appropriate IO device Coherency Plane(CP) > Forwarded to the Zambezi that owns the > A region of addressable space, unique address space for processing to that plane • DMA reads and writes are done > Each XBR only tracks addresses in its own space through cachable memory space just > Serializes accesses to coherent loads, like processor initiated memory stores and writebacks (TSO model) accesses > 40 bit Physical Address used • CP links are based on an FB-DIMM > VF Snoop links to pass transactions physical interface, but a Sun protocol • Cacheable transactions > 6.4GBps/Direction/Port – Most Memory transactions > 51.2GB/s bandwidth total • Non-Cacheable transactions > CP links are tolerant to a single lane failure – IO, Non-cachable memory, (14 lanes per link) Interrupts etc. Coherency Plane > CRC and replay features in the protocol Mem Interleave by Designator allow no-stop recovery Coherency Plane Cachable/ Non-Cachable Coherency plane Designator (512B/1GB) > Memory structures are Parity protected 39 31 30 40 Bit Physical Address 10 9 8 7 0 Sun Internal and Approved Partners Only 12
  • 13.
    UltraSPARC T2/Plus CPUCore Block Diagram • IFU – Instruction Fetch Unit > 16 KB I$, 32B lines, 8-way SA TLU IFU > 64-entry fully-associative ITLB • EXU0/1 – Integer Execution Units > 4 threads share each unit > Executes one integer instruction/cycle EXU0 EXU1 • LSU – Load/Store Unit > 8KB D$, 16B lines, 4-way SA MMU/ > 128-entry fully-associative DTLB HWTW • FGU – Floating/Graphics Unit • SPU – Stream Processing Unit > Cryptographic acceleration SPU FGU LSU • TLU – Trap Logic Unit > Updates machine state, handles exceptions and interrupts • MMU – Memory Management Unit > Hardware tablewalk (HWTW) Gasket – TLB miss handled by hardware > 8KB, 64KB, 4MB, 256MB pages xbar/L2 Sun Internal and Approved Partners Only 13
  • 14.
    Floating Point Unit • Each Core has its own Floating Point Unit. This removes completely the 2% FP restriction of T1. • Only a 6 cycle penalty to the FPU > T1 required upwards of 28 cycles • Fully-pipelined (except divide/sqrt) Data > Divide/sqrt in parallel with add or multiply operations of other threads • Full VIS 2.0 implementation in hardware, no emulation • FPU performs integer multiply, divide, population count • Multiplier incorporates enhancements for offloading modular arithmetic operations to the Crypto Unit ● ● Sun Internal and Approved Partners Only 14
  • 15.
    UltraSPARC T2 PlusCrypto Performance Absolute maximums, measured by engineering using micro-benchmarks Application throughputs will be lower than these numbers Iterations/ Block Cycles/ Function Comment Block (bytes) Block GB/S DES 2 rounds/cycle 16 rounds 8 8/9+ 12.0 3DES 2 rounds/cycle 48 rounds 8 25/24 4.0 RC4 3 overlapped cycles 1 1 37683 12.0 Data into 44 words with UltraSPARC T2 Plus AES-128K 4 words - - 32 - AES-192K 6 words into 52 words - - 37 - New Ciphers AES-256K 8 words into 60 words - - 41 - AES-128 2 column over two cycles 10 rounds 16 30/30 6.36 AES-192 2 column over two cycles 12 rounds 16 36/36 5.28 AES-256 2 column over two cycles 14 rounds 16 42/42 4.56 SHA-1 2 cycles per step 80 steps 64 160/160 4.80 SHA-256 2 cycles per step 64 steps 64 128/128 6.00 MD5 2 cycles per step 64 steps 64 128/128 6.00 CRC32c 32 bits per cycle 1 4 1 48.00 Wire Speed for UltraSPARC T2 Plus is around 5 GB/S 16,000 new SSL sessions/sec using 1024 Asymmetric Key Exchange Export restrictions apply to some of the ciphers, requires installation of the Crypto Enabling patch to use. Purchase a CD which contains the patch. Affects IPsec related ciphers, SSL typeSun Internal and Approved Partners Only acceleration is unaffected and does not need enabling. 15
  • 16.
    FB-DIMM Memory Interface Feature UltraSPARC T1 UltraSPARC T2 UltraSPARC T2 Plus Require Pins 1000 440 208 Available Bandwidth 23GBytes/second 50Gbytes/second 37.5Gbytes/second/socket Up to 64x DIMM slots Up to 32x DIMM slots/socket Memory Capacity up to 16x DIMM slots T5x20 uses 16 DIMM slots T5x40 uses 8/16 DIMM slots Data Memory Controllers 4 4 2 Memory Latency 96ns 126ns 126ns DIMM speeds 400MHz 667MHz 667MHz DIMM Power 7watts/DIMM 12.5Watts/DIMM 12.5Watts/DIMM ● ● Sun Internal and Approved Partners Only 16
  • 17.
    10/100/1000/10GbE Network Interfaces • Uses the Neptune Chip on the Motherboard which provides > Two RJ-45 ports dedicated for 10/100/1000 > Two RJ-45 ports capable of 10/100/1000 – Fitting a XAUI adapter disables the one of these ports per XAUI • Same 10 GbE XAUI cards as T5x20 server • 16 Rx and 24 Tx DMA channels for virtualization – Integrated NIU in T2 has 16 Tx channels Data – Raw UDP Tx performance on Neptune > T2 NIU – T2 processor network typically outperforms Neptune on most network benchmarks • Packet classification > Rx data can be HW classified at layer 2 ,3 and 4 – Put into the separate Rx DMA buffers by classification rules • Same 'nxge' device driver as T2 NIU. • Performance estimates, testing/tuning ongoing – 1 port 10GbE TX/RX = 9.3 Gbps with CPUs still 65-75% idle ● – 2 port 10GbE TX/RX = 12-19Gbps (better if dedicate RX/TX ports) ● Sun Internal and Approved Partners Only 17
  • 18.
    Agenda • Introduction • Comparisionof the UltraSPARC T2 to UltraSPARC VF • Sun SPARC Enterprise T5x40 servers > Overview of Features > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 2.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 18
  • 19.
    SPARC Enterprise Compute Density • 4RU chassis T5440 • 4-sockets – UltraSPARC T2 Plus Quad Socket US T2 Plus > Up to 32 cores @ 1.2GHz or 1.4Ghz > Up to 256 threads • Up to 64 FBDIMM slots > 512 GB of memory (using 8GB DIMMS) Reliable • 4 Hot-Plug 2.5” SAS disks (Raid 0,1) • DVD-RW • 4 x Redundant(2+2), hot-swappable PSUs ● Up to 256 threads / 512GB memory • 4 x Redundant, hot-swappable Fans • 2N PSU Redundancy, 1900 watts typical in a robust, redundant chassis consumption ● Taking CMT to the Mid-Range Expandable • 4 x Processor+Memory card pairs ● LDOMs consolidation dream box • 8 x8 PCIe expansion slots (2 x XAUI) • 4 Gigabit Ethernet (or up to 2 x 10GbE and 2 x1GbE) • 4 x USB 2.0 Sun Internal and Approved Partners Only 19
  • 20.
    T5440 Standard Configurations(PTO) Part Number Description 4RU 2 x 1.2GHz, 8 Core CPU, 32GB (16 X 2GB), 2x 146GB SAS 2.5”disks, 4x SEVPCJF1Z 1120W PSU, DVD-RW, 2 x Memory Expansion Boards 4 x 1.2GHz, 8 Core CPU, 64GB (32 X 2GB), 2x 146GB SAS 2.5”disks, 4x SEVPGLF1Z 1120W PSU, DVD-RW, 4 x Memory Expansion Boards 4 x 1.4GHz, 8 Core CPU, 128GB (32 X 4GB), 2x 146GB SAS 2.5”disks, SEVPHSF1Z 4x 1120W PSU, DVD-RW, 4 x Memory Expansion Boards ● All configs feature... ● 4 x 1GbE interface, ● 1 x DVD-RW slimline drive ● 8 x PCIe 8 lane slots (2 shared XAUI slots), 2 x16 mechanical ● 4 x USB 2.0 slots (2 front/2 back) ● 1 x DB9 Serial port ● 4 x Redundant, Hot-plug PSUs ● 4 x Redundant, hot-plug fans ● 2 x 146GB 2.5” SAS drives (can accommodate up to 4 drives) ● RAID 0,1 from onboard LSI 1068E controller ● 1 Year On-Site Warranty ● Rack mount kit with cable management arm ● 4 x Power Supply cables (country specific) Sun Internal and Approved Partners Only 20
  • 21.
    Sun Fire T5240/T5440Differences Feature SPARC Enterprise T5240 SPARC Enterprise T5440 Form Factor 2U, 28” deep 4U, 24.5” deep 2x UltraSPARC T2 Plus Up to 4 x UltraSPARC T2 Plus CPU 1.2/1.4 GHz 64-128 threads 1.2/1.4 GHz 128-256 threads FB-DIMM 128GB MAX FB-DIMM, 512GB MAX Memory 32x slots 64x Slots 4x GbE or 4x GbE or Network 3 x GbE + 1 x 10GbE or 3 x GbE + 1 x 10GbE or 2 x GbE + 2 x 10GbE 2 x GbE + 2 x 10GbE Internal Storage Up to 16 x 2.5” SAS, hot-swap Up to 4 x 2.5” SAS, hot-swap Removable Media 1x DVD-RW 1x DVD-RW Serial 1x RS-232, 4x USB 1x RS-232, 4x USB PCI Express slots 6x (low profile) 8x (low profile) 2x 1100W AC/2 x1200Watt DC 4 x 1120W AC Power Supply Redundant/Hot-Swap Redundant/Hot-Swap Fans 5x Redundant/Hot-Swap 4 x Redundant Hot-Swap Sun Internal and Approved Partners Only 21
  • 22.
    Front Panel Service Required LED Power/OK LED Locator LED Power Button /Button Front Fan Fault LED System Label PSU Fault LED System Overheat Fault LED Serial Number label DISK 2 DISK 3 DVD-RW DISK 0 DISK 1 Dual USB 2.0 ports Sun Internal and Approved Partners Only 22
  • 23.
    Rear Panel PCI Card Securing Latches (internal) PSU Status LED's PSU 3 PSU 2 PSU 1 PCI 0 PCI 1 PCI 2 PCI 3 PCI 4 PCI 5 PCI 6 PCI 7 PCI 0 PCI 2 PCI 3 PCI 5 PCI 6 PCI 7 PSU 0 PCI 0 = PCIe x8, mechanical x8 Quad Gigabit PCI 1 = PCIe x8, mechanical x16 SP Ethernet Ports Serial PCI 2 = PCIe x8, mechanical x8 Port SP NET 2 NET 3 Network PCI 3 = PCIe x8, mechanical x8 POSIX DB-9 Chassis Port Serial Port Status LED's Dual PCI 4 = PCIe x8, mechanical x8, or XAUI 0 Card and Locator Button USB 2.0 NET 0 NET 1 Ports PCI 5 = PCIe x8, mechanical x8, or XAUI 1 Card XAUI 1 XAUI 0 PCI 6 = PCIe x8, mechanical x16 PCI 7 = PCIe x8, mechanical x8 Sun Internal and Approved Partners Only 23
  • 24.
    Sun Fire T5440Block Diagram Mem CPU CPU Mem Mem CPU CPU Mem CPU Module = Module Module Module Module Module Module Module Module - 1 x VF CPU 0 0 2 2 1 1 3 3 - 4 x FB-DIMM Slots Mem Module = - 12 x FB-DIMM Slots Plus Plus Plus Plus T2 T2 T2 T2 x4 VRM VRM VRM VRM VRM VRM VRM VRM SAS LSI1068e x8 SAS 4x FB Links SSI SSI 4x FB Links 4x FB Links SSI SSI 4x FB Links PCIe x8 PCIe x8 PCIe x8 CL0 CL0 PCIe x8 US T2 XBR LPC CL0 0 CL0 x8 CL1 CL1 x8 x8 PLX3 PLX2 x8 US T2 XBR LPC PLX1 PLX0 PEX8548 PEX8548 CL1 CL1 PEX8548 PEX8548 1 x8 x1 x8 CL2 CL2 x8 x8 US T2 XBR Zambezi x8 x8 x8 CL2 2 LPC CL2 x8 x8 PCIe-PCI x8 CL3 CL3 XAUI 1 XAUI 0 US T2 XBR LPC Neptune CL3 3 CL3 USB host x8 QUAD GbE USB Hub USB-IDE 4x SSI PHY LPC FPGA ILOM BCM5466R PCI Card BACK FRONT BACK 0 1 2 3 panel panel panel NET SER DB-9 0 1 2 3 4 USB 2.0 USB 2.0 DVD Sun Internal and Approved Partners Only MGT MGT Quad 10/100/1000 5 6 7 24
  • 25.
    Internal View withCards installed(T5440) System Configuration 4 Fans are required Card (SCC)EEPROM MEM 3 PCIe 7 x8(m)/x8(e) PCIe 6 x16(m)/x8(e) MB/FT0 MB/FT2 MB/FT2 CPU 3 XAUI/PCIe 5 x8(m)/x8(e) CPU 1 MEM 1 Service Processor Daughter Card MEM 2 XAUI/PCIe 4 x8(m)/x8(e) CPU 2 MB/FT3 MB/FT1 MB/FT3 PCIe 3 x8(m)/x8(e) PCIe 2 x8(m)/x8(e) CPU 0 PCIe 1 x16(m)/x8(e) PCIe 0 x8(m)/x8(e) MEM 0 4 x SAS Disk Chassis Power Distribution Power Disk Board Chassis Supplies Power BUS Bars RED = +12Volts BLUE = Ground Chassis Intrusion Switch Extension to top cover from MBD Sun Internal and Approved Partners Only 25
  • 26.
    Internal View, cardsremoved(T5440) System Configuration US T2 Plus XBR Coherency Plane Chips Card (SCC)EEPROM PCIe 7 x8(m)/x8(e) MEM 3 PCIe 6 x16(m)/x8(e) MB/FT0 MB/FT2 MB/FT2 CPU 3 XAUI 1/PCIe 5 x8(m)/x8(e) CPU 1 MEM 1 Service Processor Daughter Card MEM 2 XAUI 0/PCIe 4 x8(m)/x8(e) CPU 2 MB/FT3 MB/FT1 MB/FT3 PCIe 3 x8(m)/x8(e) PCIe 2 x8(m)/x8(e) CPU 0 PCIe 1 x16(m)/x8(e) PCIe 0 x8(m)/x8(e) MEM 0 Power Distribution Power Disk Board Chassis Supplies Chassis SeperatorFence Sun Internal and Approved Partners Only 26
  • 27.
    Internal View, Motherboardonly(T5440) Neptune Quad DC-DC Converter Modules GbE/10GbE NIC MEM 3 CPU 3 MB/FT0 MB/FT2 MB/FT2 Quad GbE RJ-45 Ports CPU 1 NET MGT port Dual USB 2.0 Port MEM 1 System Status LEDs SER MGT port MEM 2 DB-9 Serial Port CPU 2 Service Processor Card Connector MB/FT3 MB/FT1 CPU 0 MEM 0 PLX PCIe Switch Chips 'Southbridge' Peripheral Chip Fan connectors, with LSI1068E Disk Controller speed control logic Sun Internal and Approved Partners Only 27
  • 28.
    Motherboard and CPU/MEMCard layout MEM 3 CPU 3 CPU 1 MEM 1 MEM 2 CPU 2 CPU 0 MEM 0 Motherboard CPU/Mem Board Memory Board Sun Internal and Approved Partners Only 28
  • 29.
    Memory Configuration Rules •1 or 4 CPU cards in a system > Each CPU card has an associated Memory Card Card/Chassis > Unpopulated Cards must be filled Top with spacers > Unpopulated DIMMs must be filled UltraSPARC T2 Plus with spacers VRM CPU • All DIMMS on a CPU & Memory VRM card pair must be the same size BR1/CH0/DO BR1/CH1/DO BANK 0 > 2GB, 4GB or 8GB only (no BR0/CH0/DO VRM Install ALL DIMMs 1GB) BR0/CH1/DO > May support Different DIMM sizes on different pairs... BR1/CH1/D2 BR1/CH1/D3 BANK 3 • 3 Configs... BR1/CH0/D2 BR1/CH0/D3 Install ALL DIMMs > Fill Bank 0 BR0/CH1/D2 BANK 2 > Fill Banks 0,1 VRM BR0/CH1/D3 Install ALL DIMMs BR0/CH0/D2 > Fill Banks 0,1,2,3 BR0/CH0/D3 BR0/CH1/D1 VRM BR0/CH0/D1 BANK 1 • All standard configs provide Install ALL DIMMs BR1/CH1/D1 BR1/CH0/D1 exactly 8 free DIMM slots simultaneously Sun Internal and Approved Partners Only 29
  • 30.
    Network Ports andOptions XAUI 0 XAUI 1 Device NET 0 NET 1 NET 2 NET 3 XAUI 0 XAUI 1 Driver nxge0 nxge1 nxge2 nxge3 nxge0* nxge1* PCI Device PCI@500/0/c/0,0 PCI@500/0/c/0,1 PCI@500/0/c/0,2 PCI@500/0/c/0,3 PCI@500/0/c/0,0 PCI@500/0/c/0,1 NOTES All Ethernet ports provided by a single Neptune Chip on the motherboard NET 2 NET 3 NET 0 and NET 1 are disabled by XAUI 1 and XAUI0,if installed, to take advantage of 10GbE networking * If a XAUI card is installed, the matching RJ-45 port is disabled (NET 0 and NET 1) XAUI cards take one of three current Fibre 10GbE transceivers Short Reach(SR), Long Reach(LR) and Long Reach Multimode(LRM) NET 0 NET 1 Copper versions of the XAUI card are expected Post-RR XAUI Cards consume the same physical space in the chassis as a PCI card, so are mutually exclusive with PCI cards in the same location Sun Internal and Approved Partners Only 30
  • 31.
    10GbE XAUI OptionCards • Interface to the Neptune chip to upgrade two of its ports to 1GbE • Cards each consume a PCI-E slot if installed • Attach to dedicated XAUI sockets on special risers • 10Gbe XAUI Card – SESY7XA1Z > Requires Fibre Tranceivers – 10GbE Short Reach(SR) – SESY7XT1Z – 10GbE Long Reach (LR) – SESY7XT2Z – 10GbE Long Reach multimode (LRM) – part No. TBD • Copper connector XAUI versions to follow post-RR Sun Internal and Approved Partners Only 31
  • 32.
    PCIe Slots Built In •No Riser Cards are used, slots are directly on the motherboard • Low Profile PCI-Express cards only • Each Slot is capable of 25Watts power • Total power capacity limited to 140Watts > Systems design point was to accommodate 8 x 17.5Watt Atlas(Quad GbE using Neptune) card • Each Slot has x8 PCIe electrical capability and at least x8 mechanical connectors • Slots 1 and 6 have x16 mechanical connectors • XAUI cards in slots 4 and 5 replace PCIe cards in those slots Sun Internal and Approved Partners Only 32
  • 33.
    PCIe External IOExpansion Unit aka Expansion Chassis, I/O Box • Up to 2 x PCI Expansion Chassis are supported > P/N : SENX8BE1Z > 4RU, 19” rack compatible , 28” deep > 1 Host card supported per US T2 Plus > Each chassis is managed by host SP • Only support PCIe card IO boats ( PCI-X ?) > Each 'boat' requires an Link Card in the host (ensign) and a link Card in the boat (captain) with special card between them > IO 'boat' supports 6 x PCIe x8 full-length, hot- pluggable PCIe cards • Chassis default configs contain > 2 x Hot-plug redundant 600Watt PSU/FAN FRU's > 1 x PCIe 'boat', 1 link card pair(host-chassis) and cable > 6 x PCIe card carriers for x8 PCIe Cards > Customers can add a further PCIe 'boat' – P/N : SENX8TE1Z and more Sun Internal and Approved Partners Only 33
  • 34.
    Disk Drives • Diskdrives are all 2.5” form factor • Original Galaxy NEMO bracket (not Marlin Bracket) • Support SAS only (Can mix SAS/SATA after qual) • SAS Drives – 73GB @ 10K RPM – 146GB @ 10K RPM – 73GB @ 15K RPM – SSD (post-RR) • Same disks as T2x00 systems Disk LED's Ready to Remove Fault Status Sun Internal and Approved Partners Only 34
  • 35.
    Disk Controller Options •T5440 has embedded LSI 1068E on the motherboard – JBOD up to 4 disks – Controls internal disks only – Unused 4 ports are not used – RAID 0 (striping) and RAID 1(mirroring) – Hot-plug supported – Hot-swap in non-RAID configs is done with 'cfgadm' • SGXPCIESAS-R-INT-Z NOT supported > Not available for T5440 due to chassis design and only 4 disk capacity • SGXPCIESAS-R-EXT-Z Supported – For external disk arrays only (no internal ports) – 8 port SAS RAID 0,1,1E,5,5EE,6,10,50,60 (depending on number of disks available) – Adaptec Card and Firmware with Intel SunRiseLake chipset – Battery Backed Write Cache (BBWC) of 256MB, at least 72 hours capacity, much longer when new Sun Internal and Approved Partners Only 35
  • 36.
    Fans and Cooling •Front to Back cooling • Two rows of fans provide redundancy > PSU fans provide sufficient redundancy for Disk cooling Back Front • Requires 4 Fans Disks PSU's – Main System Cooling MEM0 – MB/FT0 MB/FT2 MB/FT3 MB/FT3 CPU0 – MB/FT1 CPU2 Motherboard MEM2 – MB/FT2 MEM1 MB/FT1 MB/FT0 MB/FT1 – MB/FT3 CPU1 CPU3 – PSU integrated Fan Cooling flow MEM3 – Cools disk drives and PSU's – PSUFAN0 – PSUFAN1 – PSUFAN2 – PSUFAN3 Sun Internal and Approved Partners Only 36
  • 37.
    Fans modules andboards Fan Fault Amber LED Lit if fan faulty • 4 Fans Required > 60 Seconds grace if FAN removed for service • Fans are speed controlled by the Service Processor • Speeds are independantly monitored by SP • Speeds tuned to minimise vibrations > Built in vibration sensor • Speeds chosen to minimise harmonic resonance with disk drives and other fans. • Each fan has an Amber FAULT LED – if lit the fan module is faulty Sun Internal and Approved Partners Only 37
  • 38.
    Power Supplies • PSU'sare high efficiency • Output 12V (and 3.3v standby) PSU Fault • Most DC-DC conversion done in MBD LED • Integrated Fans • 4 PSU's > 2 + 2 Redundancy (Grid Redundancy) > Rated at 1120Watts each. > BEWARE... – Same form factor as SunFire X4600 PSU's ... DO NOT MIX Sun Internal and Approved Partners Only 38
  • 39.
    Rack Mounting Options •Two Rack mounting options are available.. • Standard Rack Mount (4 post) > P/N : SEVY9RK1Z > Same as Sun Fire X4600 server > Standard configs contain this kit • Tool-less Rack Kit (4 post) > P/N : SEVY9RK2Z > Optional kit, purchased seperately > Snaps into Sun and most 3rd party Racks > Requires no tools to fit or remove > Will replace tool-required version once available Sun Internal and Approved Partners Only 39
  • 40.
    Solaris for SunSPARC Enterprise T5x40 • T5x40 servers will require Solaris10 5/08 (update 5) with patches > A Patch CD is supplied in the box – Likely to be out of date, so customers should check SunSolve... • This incorporates several specific UltraSPARC T2/T2 Plus optimizations and for CMT servers in general • Also includes some specific T5x40 optimisations > UltraSPARC T2 Plus MPO optimisations > nxge driver tweaks for 10GbE devices > MMU DMA relaxed scheduling > LDOMs pre-install upgrade to version 1.0.3 (US T2 Plus support) Sun Internal and Approved Partners Only 40
  • 41.
    Solaris FMA • Nowaware of CPU Core offlining • PCIe Fabric diagnostics improved • Crypto Unit offlining • Neptune awareness (though XAUI's not diagnosed) • POST memory CE tolerance improved • FRU identity info improved in fault reports (part/ser #) • Diagnostic Engines (DE) – Existing DE have been extended for T5x40 – New Eversholt DE used for HostBridge, PCIe diagnostics • Limitations... – Can't isolate Uncorrectable Errors to a single DIMM – PCIe riser cards not in topology – XAUI card and XFP not in the topology – SouthBridge chip is not diagnosed at this time (DVD, USB,...) – IO Expansion box is not diagnosed via FMA Sun Internal and Approved Partners Only 41
  • 42.
    Agenda • Introduction • Comparisionof the UltraSPARC T2 to UltraSPARC VF • Sun SPARC Enterprise T5x40 servers > Overview of Features > System Architecture and Tours > Extra Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 3.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 42
  • 43.
    ILOM Features ata Glance • Hardware Control, Inventory, Monitoring • Host Configuration and Administration • SP Configuration and Administration • Logging, Auditing and Alerting • Authentication : – LDAP, RADIUS, Active Directory, Local Users • Services : – SSH, HTTP(S), NTP, Syslog, SMTP • Out-of-Band – Via Serial port, or dedicated Ethernet port • In-Band – Via Host OS, firmware upload, Explorer, Syslog Sun Internal and Approved Partners Only 43
  • 44.
    ILOM – IndustryStandard Manageability • ILOM enables Sun systems to be easily integrated into third party management infrastructures • Key functionality > IPMI 2.0 > SNMP V1, V2c, V3 > SSH 2.0 > LDAP, RADIUS & MS Active Directory authentication > Browser UI, SMASH DMTF CLP + ALOM style CLI • NOTE : No RKVMS functionality like Galaxy ILOM > No Java plugin required • Browsers tested include, these base versions an later releases – Mozilla 1.7 on Solaris 10, IE 6.0 on W2K Server, Firefox 1.0 and 1.5 on W2K server Sun Internal and Approved Partners Only 44
  • 45.
    ILOM – IndustryStandard Manageability • Identical to ILOM 2.0 for T5x20 • Default user is 'root' with password 'changeme' > Use for ILOM WEB GUI access and SSH access and IPMI • The ALOM 'admin' user is not defined by default, create this user with an ALOM CLI • Lost 'root' password and ILOM config is reset using jumpers > T5140/T5240 - Motherboard Jumper J10401 (short pins 1-2) > T5440 - SP Daughter card Jumper J1127 (short pins 1-2) • Note... 'sunservice' user shares the password of the 'root' user ONLY FOR SERVICE ACCESS* • Passwords must be 8 to 16 chars (used to be max 8) • Max of 10 users allowed (used to be 16) Sun Internal and Approved Partners Only 45
  • 46.
    ILOM Common Tasks •Setting up an ALOM 'admin' user from the ILOM CLI > SUNSP00144F4641BD login: root Password: changeme Waiting for daemons to initialize... Daemons ready Sun(TM) Integrated Lights Out Manager Version 2.0.4.0 Copyright 2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Warning: password is set to factory default. ->create /SP/users/admin role=Administrator cli_mode=alom Creating user... Enter new password: ******** Enter new password again: ******** Created /SP/users/admin2 -> exit SUNSP00144F4641BD login: admin Password: ******** Waiting for daemons to initialize... Daemons ready Sun(TM) Integrated Lights Out Manager Version 2.0.4.0 Copyright 2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. sc> Sun Internal and Approved Partners Only 46
  • 47.
    ILOM Common Tasks •Setting up ILOM IP configuration from the ILOM CLI > SUNSP00144F4641BD login: root Password: changeme Waiting for daemons to initialize... Daemons ready Sun(TM) Integrated Lights Out Manager Version 2.0.4.0 Copyright 2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Warning: password is set to factory default. -> set /SP/network state=enabled Set ’state’ to ’enabled’ -> set /SP/network pendingipaddress=10.8.31.163 Set ’pendingipaddress’ to ’10.8.31.163’ -> set /SP/network pendingipdiscovery=static Set ’pendingipdiscovery’ to ’static’ -> set /SP/network pendingipnetmask=255.255.252.0 Set ’pendingipnetmask’ to ’255.255.252.0’ Static IP -> set /SP/network pendingipgateway=10.8.31.248 configuration Set ’pendingipgateway’ to ’10.8.31.248’ -> set /SP/network commitpending=true Set ’commitpending’ to ’true’ -> -> set/SP/pendingipdiscovery=dhcp Set ’pendingipdiscovery’ to ’dhcp’ -> set /SP/network commitpending=true Set ’commitpending’ to ’true’ -> Dynamic IP (DHCP) configuration Sun Internal and Approved Partners Only 47
  • 48.
    ILOM IPMI functionality •IPMI is a widely used management protocol for x86 servers • ... But its new for SPARC boxes... • Download the required tools from > The Solaris Supplemental CD (an older binary package) > Latest is available at http://ipmitool.sourceforge.net – Binary packages (usually older versions) are available for Solaris x86 and SPARC, as well as Linux, Windows, Mac...etc > There is a Blueprint available from Sun also (mainly x64 centric, but applies to SPARC now too) – http://www.sun.com/blueprints/0107/820-1011.pdf -bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sdr /MB/V_+1V0_VDD | 1.01 Volts | ok /MB/V_+3V3_STBY | 3.36 Volts | ok Get the Sensor /MB/V_+3V3_MAIN | 3.36 Volts | ok Data /MB/V_+5V0_VCC | 4.97 Volts | ok Repository /MB/V_+12V0_MAIN | 12.16 Volts | ok .. .. -bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sunoem sbled set /SYS/LOCATE on /SYS/LOCATE | ON Set the System -bash-ventus2$ Locator LED flashing Sun Internal and Approved Partners Only 48
  • 49.
    ILOM Common Tasks •Logging into the ILOM Web GUI • Point Web Browser at the IP address of the ILOM > Will automatically redirect to a secure HTTPS port > Best to disable any Web Proxies for best performance > Make sure port 80, 443 are open on any firewall • Log in as any defined user, default Administration user is 'root' Sun Internal and Approved Partners Only 49
  • 50.
    ILOM/ALOM Common Tasks •A Tour of the ILOM Web GUI • GUI provides ways to set up IP address configurations for ILOM • Configure additional ILOM/ALOM users • Access Sensor Readings • Control System configurations • Carry out firmware updates. • etc.... Sun Internal and Approved Partners Only 50
  • 51.
    ALOM 2.0 vs.1.4 CLI Differences • Default CLI appearance is ILOM – User is 'root' – Must create an 'admin' Administrator user with the ALOM CLI • Flashupdate in new ALOM expects a TFTP server not an FTP server • ALOM IP address changes now active only after running ... – 'setsc netsc_commit true' – Previously changes were active only after SC reset – DHCP is enabled by default • View firmware versions with – 'showhost version' sc> showhost version Sun System Firmware 7.0.0_build_13 2007/08/10 23:44 • Only Admin and Operator roles Host flash versions: Hypervisor 1.5.0.build_13**PROTOTYPE** 2007/08/10 22:54 OBP 4.27.0.build_04***PROTOTYPE BUILD*** 2007/08/10 13:25 are defined Sun Fire[TM] Huron POST 4.27.0.build_04***PROTOTYPE BUILD*** 2007/08/10 13:51 Sun Internal and Approved Partners Only 51
  • 52.
    ALOM backward CompatibiltyCLI • Uses ALOM-CMT 1.2 for a base feature set • Includes ALOM-CMT 1.4 features... > break -d > Host Watchdog Timer > Bootmode LDOMs awareness > Forward SP events to Hot/Solaris syslog via inband channel Sun Internal and Approved Partners Only 52
  • 53.
    Agenda • Introduction • Comparisionof the UltraSPARC T2 to UltraSPARC VF • Sun SPARC Enterprise T5x40 servers > Overview of Features > Comparison to SunFire T1000/T2000/T5x20 > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 2.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 53
  • 54.
    Logical Domains 1.0.3Features • UltraSPARC T2 Plus Platform Support > 128 domains supported at RR – 256 domains work on T5440, may be qualified later, requires Solaris fixes • Feature Set very similar to 1.0.2 • I/O domain can reboot while the other domains keep running > Dependent domains would wait for I/O transactions to complete • LDoms SNMP MIB > Modeled after industry standard DMTF CIM (Common Information Model) > Running on control domain to enable remote SNMP monitoring • Domain minimization support > Minimal/reduced Solaris installation > Base minimal packages plus optional ones for guest, control, I/O or service domains Sun Internal and Approved Partners Only 54
  • 55.
    Logical Domains 1.0.3Requirements • Solaris 10 05/08 (update 5) and required patches • Systems firmware version – T5x40 Firmware 7.1.5b = SP FW 2.0.4.20+ – T5x20/SB6320 Firmware 7.0.4+ = SP FW 2.0.4.0+ – T2000/T1000/T6300/Pelton Firmware 6.5.x • SUNWldm.v LDOMs Manager Package – Version 1.0.3 Sun Internal and Approved Partners Only 55
  • 56.
    Agenda • Introduction • Comparisionof the UltraSPARC T2 to UltraSPARC VF • Sun SPARC Enterprise T5x40 servers > Overview of Features > Comparison to SunFire T1000/T2000/T5x20 > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 2.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 56
  • 57.
    Option Parts Listfor T5440(ATO) Part Number Option Description Available SEVY1CB1Z T2 Plus CPU module, 8 core, 1.2GHz, 8GB(4x2GB) with memory board RR SEVY1CC1Z T2 Plus CPU module, 8 core, 1.2GHz, 16GB(4x4GB) with memory board RR SEVY1CD1Z T2 Plus CPU module, 8 core, 1.2GHz, 32GB(4x8GB) with memory board SEVY1DD1Z T2 Plus CPU module, 8 core, 1.4GHz, 32GB(4x8GB) with memory board SEVY1DB1Z T2 Plus CPU module, 8 core, 1.4GHz, 8GB(4x2GB) with memory board Subject to 1.4GHz availability SEVY1DC1Z T2 Plus CPU module, 8 core, 1.4GHz, 16GB(4x4GB) with memory board Subject to 1.4GHz availability SESY2A1Z 2GB Memory Expansion kit(2x1GB) SESY2B2Z 4GB Memory Expansion kit(2x2GB) Non-Extended ECC version, no mixing SESY2D1Z 16GB Memory Expansion kit (2 X8 GB) SESY2C1Z 8GB Memory Expansion kit(2x4GB) SESVY3A11Z 73GB 10Krpm, 2.5” SAS Drive SESVY3C11Z 146GB 10Krpm, 2.5” SAS Drive SESVY3A21Z 73GB 15Krpm, 2.5” SAS Drive SEVY9PS51Z 1120W AC Power Supply SEVY9DV1Z DVD, 8x DVD-RW Slimline module SEVY9RK1Z Slide Rail Kit, Screw Mounting SEVY9RK2Z Slide Rail Kit, Toolless Mounting Sun Internal and Approved Partners Only 57
  • 58.
    Option Parts Listfor T5440 (cont..) Part Number Option Description Avialable Max number SG-XPCIE1FC-QF4 Qlogic, 4Gb, PCI-E Single Port RR 8 SG-XPCIE2FC-QF4 Qlogic, 4Gb, PCI-E Dual Port RR 8 SG-XPCIE1FC-EM4 Emulex, 4Gb, PCI-E Single Port RR 8 SG-XPCIE2FC-EM4 Emulex, 4Gb, PCI-E Dual Port RR 8 SG-XPCIE2SCSI-U320Z U320 SCSI, PCI-E Dual Port RR 8 SG-XPCIE8SAS-E-Z SAS, 8 Port, PCI-E RR 8 SGXCPCIESAS-R-EXT-Z SAS, 8 Port External RAID Controller RR 2 X7281A-2 Northstar Dual port GigE MMF Low profile PCI-E RR 8 X7280A-2 Northstar Dual port GigE UTP Low profile PCI-E RR 8 X1236A-Z Infiniband HBA, Dual 4x ports, PCI-E RR 8 X4447A-Z Atlas Quad Port, GigE Copper, PCI-E (up to 3/6) RR 8 X1027A-Z Atlas Dual Port, 10GigE, PCI-E RR 4 X5558A Tranceiver 10GE SR XFP (XAUI/Atlas) RR - X5560A-Z Tranceiver 10GE LR XFP (XAUI/Atlas) RR - X3000A-Z XVR-300, PCI-E x8 Graphics cards (up to 1) RR 4 SESY7XA1Z 10GbE XAUI card RR 2 TBD 10GbE XAUI Copper 10g Base-T TBD 2 X6000A Sun Crypto Accelerator 6000 RR 2 SENY8BE1Z External IO Expansion Unit – PCI-E base Config RR 2 SENY8TE1Z PCI-Express “boat” for External IO Expansion Unit (6 Hot Plug carriers) RR 4 SELY8LK1Z Copper link kit for IO Expansion unit “boat” RR 4 Sun Internal and Approved Partners Only 58
  • 59.
    Agenda • Introduction • Comparisionof the UltraSPARC T2 to UltraSPARC VF • Sun SPARC Enterprise T5x40 servers > Overview of Features > Comparison to SunFire T1000/T2000/T5x20 > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 2.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 59
  • 60.
    Tuning Out ofthe Box • Using Solaris 10 05/08 (update 5) with Patches > MPO optimisations used > DMA relaxed Ordering implemented > Default pages sizes for – Initdata Segments is 64kB pages – Private Anon, Heap, Process Stack, Text, Shared Anon segments are 4MB pages – ISM and DISM use 256MB pages – New variables introduced to control these sizes if changes are needed – max_uheap_lpsize,max_ustack_lpsize, max_privmap_lpsize, max_shm_lpsize, max_uidata_lpsize, max_utext_lpsize, max_shm_lpsize > L2 Cache hashing implemented – helps randomise accesses to L2 caches and ease thrashing > Stack biasing implemented – D-Cache is only 4 Way associative... 8 threads – Running multiple copies of a binary can lead process stacks aligning in the cache and to hot lines in the cache – Stack Bias slews the stack variable for processes. Sun Internal and Approved Partners Only 60
  • 61.
    Tuning Out ofthe Box (continued..) > Processor Groups introduced – Helps associate CPUs with some shared resources (int/FP pipelines, Crypto units etc) – Enables improved multi-level CMT load balancing and affinity policies > 'bcopy'/'uiomove' Optimizations – Key kernel operations – new implementation about 2x faster than previous • Optimisations will require patching of S10 05/08 > These will be post RR patches as they become available > Most fixes are already committed in update 5 > Check patch levels for customers with performance issues > Also check Firmware versions as this can improve performance too... Sun Internal and Approved Partners Only 61
  • 62.
    Tuning Recommendations • Lessinvolved than with T2000 • S10 5/08 is more 'automatic' – OOB optimised page size selection, thread scheduling, block copy, IP instances for zones, page coloring etc... • If tuning is necessary remember... > Improved pipeline results in Single threaded performance around 20% better > Floating Point is no longer an issue > Start with T2000 tuning options > Use updated trapstat, cpustat, • LDOMs may be an easier way to scale poorly threaded applications • Solaris Crypto framework is still being tuned > Great for big packets, working on improving small packets • A Blueprint is being prepared (Denis Sheahan) Sun Internal and Approved Partners Only 62
  • 63.
    Tuning Recommendations (cont..) •Refer to UltraSPARC T1 applications tuning best practices... > http://www.sun.com/servers/coolthreads/tnb/applications.jsp • CoolTools are being updated for UltraSPARC T2 and T2 Plus > http://www.sun.com/servers/coolthreads/overview/cooltools.jsp • Sun Studio 12 is UltraSPARC T1 and T2/T2 Plus aware > Recompiling without code changes should benefit T1/T2/T2 Plus apps • LDOMs configs can benefit significantly from a transition to T5x20 platform > Multiple IO/Service domains with higher bandwidth PCI busses > Make sure appropriate LDOMs tuning and sizing is carried out • Wider Crypto capability, so ensure code is using PKCS#11 framework > Ensure that the ncp (SSL, RSA, DSA etc) and n2cp (bulk ciphers), n2rng(random number generator) drivers are used appropriately Sun Internal and Approved Partners Only 63
  • 64.
    PCIe Performance • Easilytwo to four times the performance of T5x20 (two or four PCIe Busses) • All PCI card slots are x8 PCI electrically > Single level hierarchy > Cut through switching > PCI-E relaxed ordering for higher performance • Performance per bus... > Marketing performance = 4000MB/sec > Theoretical Rate with overheads = 3790MB/sec > Max. DMA Achievable(bidirectional) = 3096MB/sec – with 128byte packets – And 512 byte read size Sun Internal and Approved Partners Only 64
  • 65.
    Agenda • Introduction • Comparisionof the UltraSPARC T2 to UltraSPARC VF • Sun SPARC Enterprise T5x40 servers > Overview of Features > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40 • ILOM 2.0 • LDOMs for T5x40 • CRUs/FRUs and Configuration Options • Tuning and Performance • Tools, Information and References • Q & A ... Sun Internal and Approved Partners Only 65
  • 66.
    Tools and Information •Cool Tools – http://www.sun.com/servers/coolthreads/overview/cooltools.jsp • SUNWIN Tokens > Sun SPARC Enterprise T5440 – Architecture Whitepaper : 526122 – JTF : 526118 > T5440 RAS Whitepaper : TBA • Video Tours > Check the Sun.com product pages • Power Calculators > Check the Sun.com product pages Sun Internal and Approved Partners Only 66
  • 67.
    Further Information • Presto > T5440 > http://sunwebcollab.east.sun.com/gm/document-1.26.680780/Batoka_PRESTO.html • Installed Business Base (IBB) Promotions > http://sunwebcollab.east.sun.com/gm/document-1.26.680782/Batoka_IBB_Promo.html • Start Up Essentials program > http://sunwebcollab.east.sun.com/gm/document-1.26.680794/Maramba_Batoka_SU_April1.html Sun Internal and Approved Partners Only 67
  • 68.
    Reference Information • T5440Engineering TOI – http://ittdev.east.sun.com/TechTalk2/EngineeringTOIs/ • ILOM/ALOM User Guides – http://www.sun.com/products-n-solutions/hardware/docs/pdf/819-1160-13.pdf – http://www.sun.com/products-n-solutions/hardware/docs/pdf/819-7981-10.pdf • Solaris FMA – http://fma.eng – http://lightside.sfbay/n2-fma/docs.php • Sun Internal and Approved Partners Only 68
  • 69.
  • 70.
    T5x40 PCI Topology Sun Internal and Approved Partners Only 70
  • 71.
    10GbE Networking Substantiation 10GbEprice for 2 x SR-fiber PCI-E adaptors = $6,740. From HP DL380G5 pricing on HP Store, www.hp.com, 12/17/07. Cost for 2 x 10GbE and SR transceivers for Maramba and Batoka are $1,800. From sun.com, 12/17/07 Sun Internal and Approved Partners Only 71