Your SlideShare is downloading. ×
Sun sparc enterprise t5440 server technical presentation
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Sun sparc enterprise t5440 server technical presentation

1,203
views

Published on

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,203
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
34
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Sun SPARC EnterpriseT5440Rack ServersTrain-the-Trainer CoursePresenterSun Microsystems, Inc.Sun ConfidentialCDA Required 1
  • 2. NDA NotificationThis presentation contains information related to projects that are currently in planningand/or development stages. This information represents the current intentions of SunMicrosystems, Inc. However, all aspects of these projects, including, but not limited tofunding, availability, shipping dates, configurations, capacities, performance, and allother characteristics are subject to change and/or cancellation without notice.This material is confidential to Sun Microsystems and should be disclosed to non-employees only under the terms of an executed confidential-disclosure agreement.Distribution of this material to Sun Field Representatives should be done through theGSO CDA process. Under no circumstances should these roadmaps be given directly toa Sun Field Representative.The information contained herein is considered privileged, confidential and protectedfrom disclosure. If you are not the intended recipient or their employee/agentresponsible for its delivery, you are requested to refrain from any duplication ordistribution of this message or its contents, in respect of the rights of the designatedrecipient and sender and to delete it from your computer. Sun Internal and Approved Partners Only 2
  • 3. Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC T2 Plus• Sun SPARC Enterprise T5440 server > Overview of Features > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40• ILOM 2.0• LDOMs for T5x40• CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ... Sun Internal and Approved Partners Only 3
  • 4. Industrys Most Highly Threaded ServersMaximum Threading =Higher Throughput, Greater Energy & Space Efficiency T5440 Up to 256 Threads T5140/T5240 Up to 128 Threads T5120/T5220 Up to 64 Threads SB6320 US T2 blade Sun Internal and Approved Partners Only 4
  • 5. UltraSPARC T2: • Up to 8 cores @1.2GHz or 1.4GHzTrueSystem On a Chip • Up to 64 threads per CPU • Up to 16 FB-DIMMs, 4 memory controllers FB DIMM FB DIMM FB DIMM FB DIMM > Up to 64GB memory (4GB DIMMs) > 2.5x memory BW of UltraSPARC T1 FB DIMM FB DIMM FB DIMM FB DIMM • 8 x fully pipelined Floating Point units / core, 1 per core MCU MCU MCU MCU • Dual 10Gbit Ethernet and PCI-E integrated onto chip L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$ • 4MB L2 (8 banks) 16 way associative Full Cross Bar • Enhanced MAU/Security co-processor C0 C1 FPU FPU C2 FPU C3 FPU C4 FPU C5 FPU C6 FPU C7 FPU per core MAU MAU MAU MAU MAU MAU MAU MAU > DES, 3DES, AES, RC4, SHA1, DMA Sub-System SHA256, MD5, RSA to 2048 key, ECC,CRC32 NIU Sys I/F • Advanced Power saving features (E-net+) Buffer Switch Core PCI-E • 65nm process technology Power ~95W <1.5W/thread x8 @2.5GHz 2x 10GE Ethernet 4GBytes/s bi-directional Sun Internal and Approved Partners Only 5
  • 6. UltraSPARC T2 Processor SPC SPARCv9 CMT Core L2 Data L2 Data Bank 0 Bank 4 CCX Core-Cache Crossbar L2B0 SPARC SPARC SPARC SPARC L2B4 L2$ Level 2 Cache Bank L2 Data Core 0 Core 1 Core 5 Core 4 L2 Data CLC Coherence & Link Controller Bank 1 Bank 5 MCU Memory Control Unit L2B1 L2B5 SIU System Interface Unit L2 L2 L2 L2 MCU0 MCU2 NCU Non-cacheable Unit NCU TAG0 TAG1 TAG5 TAG4 FSR MAC Media Access Controller EFU MCU1 MCU3 DMU Data Management Unit SIO SII L2B3 CCX L2B7 FSR CCU PEU PCI Express Unit L2 Data L2 Data Bank 3 Bank 7 L2 L2 L2 L2 L2B2 L2B6 TAG2 TAG3 TAG7 TAG6 L2 Data L2 Data Bank 2 Bank 6 SPARC SPARC SPARC SPARC DMU Core 2 Core 3 Core 7 Core 6 RDP TDS PEU RTX PSR ESR FSR MACFor a video tour with Rick Hetherington, go to http://www.youtube.com/watch?v=B8DNexYcsa4 Sun Internal and Approved Partners Only 6
  • 7. UltraSPARC T2 Plus: • Per Socket Multi-Socket US T2 > Up to 8 cores @1.2GHz orEach Coherency Link 6.4GBytes/s per direction, 12.8GB/s total 1.4GHz 4 x Coherency links give Snoop bandwidth of 51.2GB/s > Up to 64 Threads per socket 4x Coherency > 4MB L2$ 8 banks x 16 Way SA Links > Up to 8 FPUs, 1 per core FB-DIMM FB-DIMM FB-DIMM FB-DIMM > Up to 8 Crypto cores, 1 per core FB-DIMM FB-DIMM FB-DIMM FB-DIMM > DES, 3DES, AES, RC4, SHA1, SHA256, MD5, RSA to 2048 key, ECC,CRC32 MCU MCU > 2 memory Controllers, 16 DIMMS CU CU CU CU > 25Read+13WriteGB/s Memory BW L2$ L2$ L2$ L2$ L2$ L2$ L2$ L2$ > One x8 PCIe interface Full Cross Bar • Per system CO C1 C2 C3 C4 C5 C6 C7 > 2 sockets glue-less FPU MAU FPU MAU FPU MAU FPU MAU FPU MAU FPU MAU FPU MAU FPU MAU > 4 sockets using UST2 Plus XBR > 16 or 32 PCIe lanes on 2 or 4 System I/F PCI-Express Socket systems • 65nm process technology SSI x8 @2.5GHz Bus Power ~105W 4GBytes/s <1.5W/thread bi-directional Sun Internal and Approved Partners Only 7
  • 8. UltraSPARC T2 Plus Processor SPC SPARCv9 CMT Core CCX Core-Cache Crossbar L2$ Level 2 Cache Bank CLC Coherence & Link Controller MCU Memory Control Unit SIU System Interface Unit NCU Non-cacheable Unit NCX Non-cacheable Crossbar DMU Data Management Unit PEU PCI Express Unit Sun Internal and Approved Partners Only 8
  • 9. CMT Processor Comparison UltraSPARC T1 UltraSPARC T2 UltraSPARC T2 PlusTechnology 90nm 65nm 65nm# of Cores 8 8 8# of Threads 32 64 64# of Sockets 1 1 1–4Core Frequency (Ghz) 1 – 1.4 1.2 – 1.4 1.2 – 1.4Execution pipelines per core 1 2 2Instructions per cycle per pipe 1 1 1FPUs per chip 1 8 8I$ size (per core) 16KB 16KB 16KBD$ size (per core) 8KB 8KB 8KBL2$ size/Set Association 3MB/12-Way 4MB/16-Way 4MB/16-WayMemory Type DDR2 FBDIMM-1 FBDIMM-1Memory Bandwidth (GB/s) 21.3 42READ + 21WRITE 25READ + 13WRITECoherence Bandwdth (GB/s) - - 4 x (6.4READ +6.4WRITE)IO Bus JBUS x8 PCIe x8 PCIeIO Bandwidth (GB/s) 2.5 2+2 2+2On-chip Networking - 2 x 10GE -Crypto Acceleration RSA RSA, RC4, DES, 3DES, AES, SHA1-256, MD5Power (Watts, Watts/thread) 72, 2.25 95, 1.5 105, 1.6 * Sun Internal and Approved Partners Only 9
  • 10. UltraSPARC T2 Plus 2-Socket System Coherency Links Dual Channel Dual Channel Dual Channel Dual Channel FBDIMM FBDIMM FBDIMM FBDIMM Memory Controller Memory Controller Memory Controller Memory Controller Coherence Coherence Coherence Coherence Coherence Coherence Coherence Coherence Unit Unit Unit Unit Unit Unit Unit Unit UltraSPARC T2 Plus Cores, Crossbar, L2$ UltraSPARC T2 Plus Cores, Crossbar, L2$ (8 cores, 64 threads, 4MB L2$) (8 cores, 64 threads, 4MB L2$) PCI-Express NCU, DMU NCX PCI-Express NCU, DMU NCX System IO (Network, Disk, etc.) Sun Internal and Approved Partners Only 10
  • 11. UltraSPARC T2 Plus 4-Socket System 6.4GB/s 6.4GB/s Dual Channel FBDIMM Dual Channel US T2 Plus US T2 Plus FBDIMMPCI-Express PCI-Express 8 Cores 6.4GB/s 6.4GB/s 8 Cores 6.4GB/s 6.4GB/s 64 Threads Coherence 64 Threads Dual Channel 4MB L2$ 4MB L2$ Coherence(4) Hubs FBDIMM Dual Channel FBDIMM Coherence Coherence UltraSPARC Hubs (4) Hubs (4) HubsPlus T2 (4) 6.4GB/s XBR 6.4GB/s 44 ports/hub ports/hub 6.4GB/s 4 ports/hub 6.4GB/s Dual Channel inin each dir each dir FBDIMM Dual Channel in each dir FBDIMMPCI-Express US T2 Plus VF 4 ports/hub US T2 Plus PCI-Express 6.4GB/s per port 6.4GB/s 6.4GB/s 8 Cores 8 Cores in each dir 6.4GB/s 8 Cores 6.4GB/s 64 Threads 8 Threads/Core 64 Threads Dual Channel FBDIMM 4MB L2$ 4MB L2$ Dual Channel 4MB L2$ FBDIMM 6.4GB/s 6.4GB/s Sun Internal and Approved Partners Only 11
  • 12. US T2 Plus XBR Coherency Hub Chips... • 4 XBR chips in a system • XBR routes non-cacheable PIOs to the • Each operates in its own appropriate IO device Coherency Plane(CP) > Forwarded to the Zambezi that owns the > A region of addressable space, unique address space for processing to that plane • DMA reads and writes are done > Each XBR only tracks addresses in its own space through cachable memory space just > Serializes accesses to coherent loads, like processor initiated memory stores and writebacks (TSO model) accesses > 40 bit Physical Address used • CP links are based on an FB-DIMM > VF Snoop links to pass transactions physical interface, but a Sun protocol • Cacheable transactions > 6.4GBps/Direction/Port – Most Memory transactions > 51.2GB/s bandwidth total • Non-Cacheable transactions > CP links are tolerant to a single lane failure – IO, Non-cachable memory, (14 lanes per link) Interrupts etc. Coherency Plane > CRC and replay features in the protocol Mem Interleave by Designator allow no-stop recovery Coherency Plane Cachable/Non-Cachable Coherency plane Designator (512B/1GB) > Memory structures are Parity protected39 31 30 40 Bit Physical Address 10 9 8 7 0 Sun Internal and Approved Partners Only 12
  • 13. UltraSPARC T2/Plus CPU Core Block Diagram • IFU – Instruction Fetch Unit > 16 KB I$, 32B lines, 8-way SA TLU IFU > 64-entry fully-associative ITLB • EXU0/1 – Integer Execution Units > 4 threads share each unit > Executes one integer instruction/cycle EXU0 EXU1 • LSU – Load/Store Unit > 8KB D$, 16B lines, 4-way SA MMU/ > 128-entry fully-associative DTLB HWTW • FGU – Floating/Graphics Unit • SPU – Stream Processing Unit > Cryptographic acceleration SPU FGU LSU • TLU – Trap Logic Unit > Updates machine state, handles exceptions and interrupts • MMU – Memory Management Unit > Hardware tablewalk (HWTW) Gasket – TLB miss handled by hardware > 8KB, 64KB, 4MB, 256MB pages xbar/L2 Sun Internal and Approved Partners Only 13
  • 14. Floating Point Unit • Each Core has its own Floating Point Unit. This removes completely the 2% FP restriction of T1. • Only a 6 cycle penalty to the FPU > T1 required upwards of 28 cycles • Fully-pipelined (except divide/sqrt) Data > Divide/sqrt in parallel with add or multiply operations of other threads • Full VIS 2.0 implementation in hardware, no emulation • FPU performs integer multiply, divide, population count • Multiplier incorporates enhancements for offloading modular arithmetic operations to the Crypto Unit●● Sun Internal and Approved Partners Only 14
  • 15. UltraSPARC T2 Plus Crypto Performance Absolute maximums, measured by engineering using micro-benchmarks Application throughputs will be lower than these numbers Iterations/ Block Cycles/ Function Comment Block (bytes) Block GB/SDES 2 rounds/cycle 16 rounds 8 8/9+ 12.03DES 2 rounds/cycle 48 rounds 8 25/24 4.0RC4 3 overlapped cycles 1 1 37683 12.0 Data into 44 words with UltraSPARC T2 PlusAES-128K 4 words - - 32 -AES-192K 6 words into 52 words - - 37 - New CiphersAES-256K 8 words into 60 words - - 41 -AES-128 2 column over two cycles 10 rounds 16 30/30 6.36AES-192 2 column over two cycles 12 rounds 16 36/36 5.28AES-256 2 column over two cycles 14 rounds 16 42/42 4.56SHA-1 2 cycles per step 80 steps 64 160/160 4.80SHA-256 2 cycles per step 64 steps 64 128/128 6.00MD5 2 cycles per step 64 steps 64 128/128 6.00CRC32c 32 bits per cycle 1 4 1 48.00 Wire Speed for UltraSPARC T2 Plus is around 5 GB/S 16,000 new SSL sessions/sec using 1024 Asymmetric Key Exchange Export restrictions apply to some of the ciphers, requires installation of the Crypto Enabling patch to use. Purchase a CD which contains the patch.Affects IPsec related ciphers, SSL typeSun Internal and Approved Partners Only acceleration is unaffected and does not need enabling. 15
  • 16. FB-DIMM Memory Interface Feature UltraSPARC T1 UltraSPARC T2 UltraSPARC T2 Plus Require Pins 1000 440 208 Available Bandwidth 23GBytes/second 50Gbytes/second 37.5Gbytes/second/socket Up to 64x DIMM slots Up to 32x DIMM slots/socket Memory Capacity up to 16x DIMM slots T5x20 uses 16 DIMM slots T5x40 uses 8/16 DIMM slots Data Memory Controllers 4 4 2 Memory Latency 96ns 126ns 126ns DIMM speeds 400MHz 667MHz 667MHz DIMM Power 7watts/DIMM 12.5Watts/DIMM 12.5Watts/DIMM●● Sun Internal and Approved Partners Only 16
  • 17. 10/100/1000/10GbE Network Interfaces • Uses the Neptune Chip on the Motherboard which provides > Two RJ-45 ports dedicated for 10/100/1000 > Two RJ-45 ports capable of 10/100/1000 – Fitting a XAUI adapter disables the one of these ports per XAUI • Same 10 GbE XAUI cards as T5x20 server • 16 Rx and 24 Tx DMA channels for virtualization – Integrated NIU in T2 has 16 Tx channels Data – Raw UDP Tx performance on Neptune > T2 NIU – T2 processor network typically outperforms Neptune on most network benchmarks • Packet classification > Rx data can be HW classified at layer 2 ,3 and 4 – Put into the separate Rx DMA buffers by classification rules • Same nxge device driver as T2 NIU. • Performance estimates, testing/tuning ongoing – 1 port 10GbE TX/RX = 9.3 Gbps with CPUs still 65-75% idle● – 2 port 10GbE TX/RX = 12-19Gbps (better if dedicate RX/TX ports)● Sun Internal and Approved Partners Only 17
  • 18. Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC VF• Sun SPARC Enterprise T5x40 servers > Overview of Features > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40• ILOM 2.0• LDOMs for T5x40• CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ... Sun Internal and Approved Partners Only 18
  • 19. SPARC Enterprise Compute Density • 4RU chassis T5440 • 4-sockets – UltraSPARC T2 PlusQuad Socket US T2 Plus > Up to 32 cores @ 1.2GHz or 1.4Ghz > Up to 256 threads • Up to 64 FBDIMM slots > 512 GB of memory (using 8GB DIMMS) Reliable • 4 Hot-Plug 2.5” SAS disks (Raid 0,1) • DVD-RW • 4 x Redundant(2+2), hot-swappable PSUs● Up to 256 threads / 512GB memory • 4 x Redundant, hot-swappable Fans • 2N PSU Redundancy, 1900 watts typical in a robust, redundant chassis consumption● Taking CMT to the Mid-Range Expandable • 4 x Processor+Memory card pairs● LDOMs consolidation dream box • 8 x8 PCIe expansion slots (2 x XAUI) • 4 Gigabit Ethernet (or up to 2 x 10GbE and 2 x1GbE) • 4 x USB 2.0 Sun Internal and Approved Partners Only 19
  • 20. T5440 Standard Configurations (PTO) Part Number Description 4RU 2 x 1.2GHz, 8 Core CPU, 32GB (16 X 2GB), 2x 146GB SAS 2.5”disks, 4x SEVPCJF1Z 1120W PSU, DVD-RW, 2 x Memory Expansion Boards 4 x 1.2GHz, 8 Core CPU, 64GB (32 X 2GB), 2x 146GB SAS 2.5”disks, 4x SEVPGLF1Z 1120W PSU, DVD-RW, 4 x Memory Expansion Boards 4 x 1.4GHz, 8 Core CPU, 128GB (32 X 4GB), 2x 146GB SAS 2.5”disks, SEVPHSF1Z 4x 1120W PSU, DVD-RW, 4 x Memory Expansion Boards ● All configs feature... ● 4 x 1GbE interface, ● 1 x DVD-RW slimline drive ● 8 x PCIe 8 lane slots (2 shared XAUI slots), 2 x16 mechanical ● 4 x USB 2.0 slots (2 front/2 back) ● 1 x DB9 Serial port ● 4 x Redundant, Hot-plug PSUs ● 4 x Redundant, hot-plug fans ● 2 x 146GB 2.5” SAS drives (can accommodate up to 4 drives) ● RAID 0,1 from onboard LSI 1068E controller ● 1 Year On-Site Warranty ● Rack mount kit with cable management arm ● 4 x Power Supply cables (country specific) Sun Internal and Approved Partners Only 20
  • 21. Sun Fire T5240/T5440 Differences Feature SPARC Enterprise T5240 SPARC Enterprise T5440 Form Factor 2U, 28” deep 4U, 24.5” deep 2x UltraSPARC T2 Plus Up to 4 x UltraSPARC T2 Plus CPU 1.2/1.4 GHz 64-128 threads 1.2/1.4 GHz 128-256 threads FB-DIMM 128GB MAX FB-DIMM, 512GB MAX Memory 32x slots 64x Slots 4x GbE or 4x GbE or Network 3 x GbE + 1 x 10GbE or 3 x GbE + 1 x 10GbE or 2 x GbE + 2 x 10GbE 2 x GbE + 2 x 10GbE Internal Storage Up to 16 x 2.5” SAS, hot-swap Up to 4 x 2.5” SAS, hot-swap Removable Media 1x DVD-RW 1x DVD-RW Serial 1x RS-232, 4x USB 1x RS-232, 4x USB PCI Express slots 6x (low profile) 8x (low profile) 2x 1100W AC/2 x1200Watt DC 4 x 1120W AC Power Supply Redundant/Hot-Swap Redundant/Hot-Swap Fans 5x Redundant/Hot-Swap 4 x Redundant Hot-Swap Sun Internal and Approved Partners Only 21
  • 22. Front Panel Service Required LED Power/OK LEDLocator LED Power Button /Button Front Fan Fault LED System Label PSU Fault LED System Overheat Fault LED Serial Number label DISK 2 DISK 3 DVD-RW DISK 0 DISK 1 Dual USB 2.0 ports Sun Internal and Approved Partners Only 22
  • 23. Rear Panel PCI Card Securing Latches (internal)PSUStatusLEDs PSU 3 PSU 2 PSU 1 PCI 0 PCI 1 PCI 2 PCI 3 PCI 4 PCI 5 PCI 6 PCI 7 PCI 0 PCI 2 PCI 3 PCI 5 PCI 6 PCI 7 PSU 0PCI 0 = PCIe x8, mechanical x8 Quad GigabitPCI 1 = PCIe x8, mechanical x16 SP Ethernet Ports SerialPCI 2 = PCIe x8, mechanical x8 Port SP NET 2 NET 3 NetworkPCI 3 = PCIe x8, mechanical x8 POSIX DB-9 Chassis Port Serial Port Status LEDs DualPCI 4 = PCIe x8, mechanical x8, or XAUI 0 Card and Locator Button USB 2.0 NET 0 NET 1 PortsPCI 5 = PCIe x8, mechanical x8, or XAUI 1 Card XAUI 1 XAUI 0PCI 6 = PCIe x8, mechanical x16PCI 7 = PCIe x8, mechanical x8 Sun Internal and Approved Partners Only 23
  • 24. Sun Fire T5440 Block Diagram Mem CPU CPU Mem Mem CPU CPU Mem CPU Module = Module Module Module Module Module Module Module Module - 1 x VF CPU 0 0 2 2 1 1 3 3 - 4 x FB-DIMM Slots Mem Module = - 12 x FB-DIMM Slots Plus Plus Plus Plus T2 T2 T2 T2 x4 VRM VRM VRM VRM VRM VRM VRM VRM SAS LSI1068e x8 SAS 4x FB Links SSI SSI 4x FB Links 4x FB Links SSI SSI 4x FB Links PCIe x8 PCIe x8 PCIe x8 CL0 CL0 PCIe x8 US T2 XBR LPC CL0 0 CL0 x8 CL1 CL1 x8 x8 PLX3 PLX2 x8 US T2 XBR LPC PLX1 PLX0 PEX8548 PEX8548 CL1 CL1 PEX8548 PEX8548 1x8 x1 x8 CL2 CL2 x8 x8 US T2 XBR Zambezi x8 x8 x8 CL2 2 LPC CL2 x8 x8 PCIe-PCI x8 CL3 CL3 XAUI 1 XAUI 0 US T2 XBR LPC Neptune CL3 3 CL3 USB host x8 QUAD GbE USB Hub USB-IDE 4x SSI PHY LPC FPGA ILOM BCM5466R PCI Card BACK FRONT BACK 0 1 2 3 panel panel panel NET SER DB-9 0 1 2 3 4 USB 2.0 USB 2.0 DVD Sun Internal and Approved Partners Only MGT MGT Quad 10/100/1000 5 6 7 24
  • 25. Internal View with Cards installed(T5440) System Configuration 4 Fans are required Card (SCC)EEPROM MEM 3 PCIe 7 x8(m)/x8(e) PCIe 6 x16(m)/x8(e) MB/FT0 MB/FT2 MB/FT2 CPU 3 XAUI/PCIe 5 x8(m)/x8(e) CPU 1 MEM 1 Service Processor Daughter Card MEM 2 XAUI/PCIe 4 x8(m)/x8(e) CPU 2 MB/FT3 MB/FT1 MB/FT3 PCIe 3 x8(m)/x8(e) PCIe 2 x8(m)/x8(e) CPU 0 PCIe 1 x16(m)/x8(e) PCIe 0 x8(m)/x8(e) MEM 0 4 x SASDisk Chassis Power Distribution Power Disk Board Chassis Supplies Power BUS Bars RED = +12Volts BLUE = Ground Chassis Intrusion Switch Extension to top cover from MBD Sun Internal and Approved Partners Only 25
  • 26. Internal View, cards removed(T5440) System Configuration US T2 Plus XBR Coherency Plane Chips Card (SCC)EEPROM PCIe 7 x8(m)/x8(e) MEM 3 PCIe 6 x16(m)/x8(e) MB/FT0 MB/FT2 MB/FT2 CPU 3 XAUI 1/PCIe 5 x8(m)/x8(e) CPU 1 MEM 1 Service Processor Daughter Card MEM 2 XAUI 0/PCIe 4 x8(m)/x8(e) CPU 2 MB/FT3 MB/FT1 MB/FT3 PCIe 3 x8(m)/x8(e) PCIe 2 x8(m)/x8(e) CPU 0 PCIe 1 x16(m)/x8(e) PCIe 0 x8(m)/x8(e) MEM 0 Power Distribution Power Disk Board Chassis Supplies Chassis SeperatorFence Sun Internal and Approved Partners Only 26
  • 27. Internal View, Motherboard only(T5440) Neptune Quad DC-DC Converter Modules GbE/10GbE NIC MEM 3 CPU 3 MB/FT0 MB/FT2 MB/FT2 Quad GbE RJ-45 Ports CPU 1 NET MGT port Dual USB 2.0 Port MEM 1 System Status LEDs SER MGT port MEM 2 DB-9 Serial Port CPU 2 Service Processor Card Connector MB/FT3 MB/FT1 CPU 0 MEM 0 PLX PCIe Switch Chips Southbridge Peripheral ChipFan connectors, with LSI1068E Disk Controller speed control logic Sun Internal and Approved Partners Only 27
  • 28. Motherboard and CPU/MEM Card layout MEM 3 CPU 3 CPU 1 MEM 1 MEM 2 CPU 2 CPU 0 MEM 0 Motherboard CPU/Mem Board Memory Board Sun Internal and Approved Partners Only 28
  • 29. Memory Configuration Rules• 1 or 4 CPU cards in a system > Each CPU card has an associated Memory Card Card/Chassis > Unpopulated Cards must be filled Top with spacers > Unpopulated DIMMs must be filled UltraSPARC T2 Plus with spacers VRM CPU• All DIMMS on a CPU & Memory VRM card pair must be the same size BR1/CH0/DO BR1/CH1/DO BANK 0 > 2GB, 4GB or 8GB only (no BR0/CH0/DO VRM Install ALL DIMMs 1GB) BR0/CH1/DO > May support Different DIMM sizes on different pairs... BR1/CH1/D2 BR1/CH1/D3 BANK 3• 3 Configs... BR1/CH0/D2 BR1/CH0/D3 Install ALL DIMMs > Fill Bank 0 BR0/CH1/D2 BANK 2 > Fill Banks 0,1 VRM BR0/CH1/D3 Install ALL DIMMs BR0/CH0/D2 > Fill Banks 0,1,2,3 BR0/CH0/D3 BR0/CH1/D1 VRM BR0/CH0/D1 BANK 1• All standard configs provide Install ALL DIMMs BR1/CH1/D1 BR1/CH0/D1 exactly 8 free DIMM slots simultaneously Sun Internal and Approved Partners Only 29
  • 30. Network Ports and Options XAUI 0 XAUI 1 Device NET 0 NET 1 NET 2 NET 3 XAUI 0 XAUI 1 Driver nxge0 nxge1 nxge2 nxge3 nxge0* nxge1* PCI Device PCI@500/0/c/0,0 PCI@500/0/c/0,1 PCI@500/0/c/0,2 PCI@500/0/c/0,3 PCI@500/0/c/0,0 PCI@500/0/c/0,1 NOTES All Ethernet ports provided by a single Neptune Chip on the motherboard NET 2 NET 3NET 0 and NET 1 are disabled by XAUI 1 and XAUI0,if installed, to take advantage of 10GbE networking * If a XAUI card is installed, the matching RJ-45 port is disabled (NET 0 and NET 1) XAUI cards take one of three current Fibre 10GbE transceivers Short Reach(SR), Long Reach(LR) and Long Reach Multimode(LRM) NET 0 NET 1 Copper versions of the XAUI card are expected Post-RR XAUI Cards consume the same physical space in the chassis as a PCI card, so are mutually exclusive with PCI cards in the same location Sun Internal and Approved Partners Only 30
  • 31. 10GbE XAUI Option Cards• Interface to the Neptune chip to upgrade two of its ports to 1GbE• Cards each consume a PCI-E slot if installed• Attach to dedicated XAUI sockets on special risers• 10Gbe XAUI Card – SESY7XA1Z > Requires Fibre Tranceivers – 10GbE Short Reach(SR) – SESY7XT1Z – 10GbE Long Reach (LR) – SESY7XT2Z – 10GbE Long Reach multimode (LRM) – part No. TBD• Copper connector XAUI versions to follow post-RR Sun Internal and Approved Partners Only 31
  • 32. PCIe SlotsBuilt In• No Riser Cards are used, slots are directly on the motherboard• Low Profile PCI-Express cards only• Each Slot is capable of 25Watts power• Total power capacity limited to 140Watts > Systems design point was to accommodate 8 x 17.5Watt Atlas(Quad GbE using Neptune) card• Each Slot has x8 PCIe electrical capability and at least x8 mechanical connectors• Slots 1 and 6 have x16 mechanical connectors• XAUI cards in slots 4 and 5 replace PCIe cards in those slots Sun Internal and Approved Partners Only 32
  • 33. PCIe External IO Expansion Unit aka Expansion Chassis, I/O Box• Up to 2 x PCI Expansion Chassis are supported > P/N : SENX8BE1Z > 4RU, 19” rack compatible , 28” deep > 1 Host card supported per US T2 Plus > Each chassis is managed by host SP• Only support PCIe card IO boats ( PCI-X ?) > Each boat requires an Link Card in the host (ensign) and a link Card in the boat (captain) with special card between them > IO boat supports 6 x PCIe x8 full-length, hot- pluggable PCIe cards• Chassis default configs contain > 2 x Hot-plug redundant 600Watt PSU/FAN FRUs > 1 x PCIe boat, 1 link card pair(host-chassis) and cable > 6 x PCIe card carriers for x8 PCIe Cards > Customers can add a further PCIe boat – P/N : SENX8TE1Z and more Sun Internal and Approved Partners Only 33
  • 34. Disk Drives• Disk drives are all 2.5” form factor• Original Galaxy NEMO bracket (not Marlin Bracket)• Support SAS only (Can mix SAS/SATA after qual)• SAS Drives – 73GB @ 10K RPM – 146GB @ 10K RPM – 73GB @ 15K RPM – SSD (post-RR)• Same disks as T2x00 systems Disk LEDs Ready to Remove Fault Status Sun Internal and Approved Partners Only 34
  • 35. Disk Controller Options• T5440 has embedded LSI 1068E on the motherboard – JBOD up to 4 disks – Controls internal disks only – Unused 4 ports are not used – RAID 0 (striping) and RAID 1(mirroring) – Hot-plug supported – Hot-swap in non-RAID configs is done with cfgadm• SGXPCIESAS-R-INT-Z NOT supported > Not available for T5440 due to chassis design and only 4 disk capacity• SGXPCIESAS-R-EXT-Z Supported – For external disk arrays only (no internal ports) – 8 port SAS RAID 0,1,1E,5,5EE,6,10,50,60 (depending on number of disks available) – Adaptec Card and Firmware with Intel SunRiseLake chipset – Battery Backed Write Cache (BBWC) of 256MB, at least 72 hours capacity, much longer when new Sun Internal and Approved Partners Only 35
  • 36. Fans and Cooling• Front to Back cooling• Two rows of fans provide redundancy > PSU fans provide sufficient redundancy for Disk cooling Back Front• Requires 4 Fans Disks PSUs – Main System Cooling MEM0 – MB/FT0 MB/FT2 MB/FT3 MB/FT3 CPU0 – MB/FT1 CPU2 Motherboard MEM2 – MB/FT2 MEM1 MB/FT1 MB/FT0 MB/FT1 – MB/FT3 CPU1 CPU3 – PSU integrated Fan Cooling flow MEM3 – Cools disk drives and PSUs – PSUFAN0 – PSUFAN1 – PSUFAN2 – PSUFAN3 Sun Internal and Approved Partners Only 36
  • 37. Fans modules and boards Fan Fault Amber LED Lit if fan faulty• 4 Fans Required > 60 Seconds grace if FAN removed for service• Fans are speed controlled by the Service Processor• Speeds are independantly monitored by SP• Speeds tuned to minimise vibrations > Built in vibration sensor• Speeds chosen to minimise harmonic resonance with disk drives and other fans.• Each fan has an Amber FAULT LED – if lit the fan module is faulty Sun Internal and Approved Partners Only 37
  • 38. Power Supplies• PSUs are high efficiency• Output 12V (and 3.3v standby) PSU Fault• Most DC-DC conversion done in MBD LED• Integrated Fans• 4 PSUs > 2 + 2 Redundancy (Grid Redundancy) > Rated at 1120Watts each. > BEWARE... – Same form factor as SunFire X4600 PSUs ... DO NOT MIX Sun Internal and Approved Partners Only 38
  • 39. Rack Mounting Options• Two Rack mounting options are available..• Standard Rack Mount (4 post) > P/N : SEVY9RK1Z > Same as Sun Fire X4600 server > Standard configs contain this kit• Tool-less Rack Kit (4 post) > P/N : SEVY9RK2Z > Optional kit, purchased seperately > Snaps into Sun and most 3rd party Racks > Requires no tools to fit or remove > Will replace tool-required version once available Sun Internal and Approved Partners Only 39
  • 40. Solaris for Sun SPARC Enterprise T5x40• T5x40 servers will require Solaris10 5/08 (update 5) with patches > A Patch CD is supplied in the box – Likely to be out of date, so customers should check SunSolve...• This incorporates several specific UltraSPARC T2/T2 Plus optimizations and for CMT servers in general• Also includes some specific T5x40 optimisations > UltraSPARC T2 Plus MPO optimisations > nxge driver tweaks for 10GbE devices > MMU DMA relaxed scheduling > LDOMs pre-install upgrade to version 1.0.3 (US T2 Plus support) Sun Internal and Approved Partners Only 40
  • 41. Solaris FMA• Now aware of CPU Core offlining• PCIe Fabric diagnostics improved• Crypto Unit offlining• Neptune awareness (though XAUIs not diagnosed)• POST memory CE tolerance improved• FRU identity info improved in fault reports (part/ser #)• Diagnostic Engines (DE) – Existing DE have been extended for T5x40 – New Eversholt DE used for HostBridge, PCIe diagnostics• Limitations... – Cant isolate Uncorrectable Errors to a single DIMM – PCIe riser cards not in topology – XAUI card and XFP not in the topology – SouthBridge chip is not diagnosed at this time (DVD, USB,...) – IO Expansion box is not diagnosed via FMA Sun Internal and Approved Partners Only 41
  • 42. Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC VF• Sun SPARC Enterprise T5x40 servers > Overview of Features > System Architecture and Tours > Extra Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40• ILOM 3.0• LDOMs for T5x40• CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ... Sun Internal and Approved Partners Only 42
  • 43. ILOM Features at a Glance• Hardware Control, Inventory, Monitoring• Host Configuration and Administration• SP Configuration and Administration• Logging, Auditing and Alerting• Authentication : – LDAP, RADIUS, Active Directory, Local Users• Services : – SSH, HTTP(S), NTP, Syslog, SMTP• Out-of-Band – Via Serial port, or dedicated Ethernet port• In-Band – Via Host OS, firmware upload, Explorer, Syslog Sun Internal and Approved Partners Only 43
  • 44. ILOM – Industry Standard Manageability • ILOM enables Sun systems to be easily integrated into third party management infrastructures • Key functionality > IPMI 2.0 > SNMP V1, V2c, V3 > SSH 2.0 > LDAP, RADIUS & MS Active Directory authentication > Browser UI, SMASH DMTF CLP + ALOM style CLI • NOTE : No RKVMS functionality like Galaxy ILOM > No Java plugin required • Browsers tested include, these base versions an later releases – Mozilla 1.7 on Solaris 10, IE 6.0 on W2K Server, Firefox 1.0 and 1.5 on W2K server Sun Internal and Approved Partners Only 44
  • 45. ILOM – Industry Standard Manageability • Identical to ILOM 2.0 for T5x20 • Default user is root with password changeme > Use for ILOM WEB GUI access and SSH access and IPMI • The ALOM admin user is not defined by default, create this user with an ALOM CLI • Lost root password and ILOM config is reset using jumpers > T5140/T5240 - Motherboard Jumper J10401 (short pins 1-2) > T5440 - SP Daughter card Jumper J1127 (short pins 1-2) • Note... sunservice user shares the password of the root user ONLY FOR SERVICE ACCESS* • Passwords must be 8 to 16 chars (used to be max 8) • Max of 10 users allowed (used to be 16) Sun Internal and Approved Partners Only 45
  • 46. ILOM Common Tasks• Setting up an ALOM admin user from the ILOM CLI > SUNSP00144F4641BD login: root Password: changeme Waiting for daemons to initialize... Daemons ready Sun(TM) Integrated Lights Out Manager Version 2.0.4.0 Copyright 2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Warning: password is set to factory default. ->create /SP/users/admin role=Administrator cli_mode=alom Creating user... Enter new password: ******** Enter new password again: ******** Created /SP/users/admin2 -> exit SUNSP00144F4641BD login: admin Password: ******** Waiting for daemons to initialize... Daemons ready Sun(TM) Integrated Lights Out Manager Version 2.0.4.0 Copyright 2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. sc> Sun Internal and Approved Partners Only 46
  • 47. ILOM Common Tasks• Setting up ILOM IP configuration from the ILOM CLI > SUNSP00144F4641BD login: root Password: changeme Waiting for daemons to initialize... Daemons ready Sun(TM) Integrated Lights Out Manager Version 2.0.4.0 Copyright 2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Warning: password is set to factory default. -> set /SP/network state=enabled Set ’state’ to ’enabled’ -> set /SP/network pendingipaddress=10.8.31.163 Set ’pendingipaddress’ to ’10.8.31.163’ -> set /SP/network pendingipdiscovery=static Set ’pendingipdiscovery’ to ’static’ -> set /SP/network pendingipnetmask=255.255.252.0 Set ’pendingipnetmask’ to ’255.255.252.0’ Static IP -> set /SP/network pendingipgateway=10.8.31.248 configuration Set ’pendingipgateway’ to ’10.8.31.248’ -> set /SP/network commitpending=true Set ’commitpending’ to ’true’ -> -> set/SP/pendingipdiscovery=dhcp Set ’pendingipdiscovery’ to ’dhcp’ -> set /SP/network commitpending=true Set ’commitpending’ to ’true’ -> Dynamic IP (DHCP) configuration Sun Internal and Approved Partners Only 47
  • 48. ILOM IPMI functionality• IPMI is a widely used management protocol for x86 servers• ... But its new for SPARC boxes...• Download the required tools from > The Solaris Supplemental CD (an older binary package) > Latest is available at http://ipmitool.sourceforge.net – Binary packages (usually older versions) are available for Solaris x86 and SPARC, as well as Linux, Windows, Mac...etc > There is a Blueprint available from Sun also (mainly x64 centric, but applies to SPARC now too) – http://www.sun.com/blueprints/0107/820-1011.pdf -bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sdr /MB/V_+1V0_VDD | 1.01 Volts | ok /MB/V_+3V3_STBY | 3.36 Volts | ok Get the Sensor /MB/V_+3V3_MAIN | 3.36 Volts | ok Data /MB/V_+5V0_VCC | 4.97 Volts | ok Repository /MB/V_+12V0_MAIN | 12.16 Volts | ok .. .. -bash-ventus2$ /opt/ipmitool/bin/ipmitool -U root -P changeme -H 10.10.10.10 sunoem sbled set /SYS/LOCATE on /SYS/LOCATE | ON Set the System -bash-ventus2$ Locator LED flashing Sun Internal and Approved Partners Only 48
  • 49. ILOM Common Tasks• Logging into the ILOM Web GUI • Point Web Browser at the IP address of the ILOM > Will automatically redirect to a secure HTTPS port > Best to disable any Web Proxies for best performance > Make sure port 80, 443 are open on any firewall • Log in as any defined user, default Administration user is root Sun Internal and Approved Partners Only 49
  • 50. ILOM/ALOM Common Tasks• A Tour of the ILOM Web GUI • GUI provides ways to set up IP address configurations for ILOM • Configure additional ILOM/ALOM users • Access Sensor Readings • Control System configurations • Carry out firmware updates. • etc.... Sun Internal and Approved Partners Only 50
  • 51. ALOM 2.0 vs. 1.4 CLI Differences• Default CLI appearance is ILOM – User is root – Must create an admin Administrator user with the ALOM CLI• Flashupdate in new ALOM expects a TFTP server not an FTP server• ALOM IP address changes now active only after running ... – setsc netsc_commit true – Previously changes were active only after SC reset – DHCP is enabled by default• View firmware versions with – showhost version sc> showhost version Sun System Firmware 7.0.0_build_13 2007/08/10 23:44• Only Admin and Operator roles Host flash versions: Hypervisor 1.5.0.build_13**PROTOTYPE** 2007/08/10 22:54 OBP 4.27.0.build_04***PROTOTYPE BUILD*** 2007/08/10 13:25 are defined Sun Fire[TM] Huron POST 4.27.0.build_04***PROTOTYPE BUILD*** 2007/08/10 13:51 Sun Internal and Approved Partners Only 51
  • 52. ALOM backward Compatibilty CLI• Uses ALOM-CMT 1.2 for a base feature set• Includes ALOM-CMT 1.4 features... > break -d > Host Watchdog Timer > Bootmode LDOMs awareness > Forward SP events to Hot/Solaris syslog via inband channel Sun Internal and Approved Partners Only 52
  • 53. Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC VF• Sun SPARC Enterprise T5x40 servers > Overview of Features > Comparison to SunFire T1000/T2000/T5x20 > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40• ILOM 2.0• LDOMs for T5x40• CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ... Sun Internal and Approved Partners Only 53
  • 54. Logical Domains 1.0.3 Features• UltraSPARC T2 Plus Platform Support > 128 domains supported at RR – 256 domains work on T5440, may be qualified later, requires Solaris fixes• Feature Set very similar to 1.0.2• I/O domain can reboot while the other domains keep running > Dependent domains would wait for I/O transactions to complete• LDoms SNMP MIB > Modeled after industry standard DMTF CIM (Common Information Model) > Running on control domain to enable remote SNMP monitoring• Domain minimization support > Minimal/reduced Solaris installation > Base minimal packages plus optional ones for guest, control, I/O or service domains Sun Internal and Approved Partners Only 54
  • 55. Logical Domains 1.0.3 Requirements • Solaris 10 05/08 (update 5) and required patches • Systems firmware version – T5x40 Firmware 7.1.5b = SP FW 2.0.4.20+ – T5x20/SB6320 Firmware 7.0.4+ = SP FW 2.0.4.0+ – T2000/T1000/T6300/Pelton Firmware 6.5.x • SUNWldm.v LDOMs Manager Package – Version 1.0.3 Sun Internal and Approved Partners Only 55
  • 56. Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC VF• Sun SPARC Enterprise T5x40 servers > Overview of Features > Comparison to SunFire T1000/T2000/T5x20 > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40• ILOM 2.0• LDOMs for T5x40• CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ... Sun Internal and Approved Partners Only 56
  • 57. Option Parts List for T5440(ATO) Part Number Option Description Available SEVY1CB1Z T2 Plus CPU module, 8 core, 1.2GHz, 8GB(4x2GB) with memory board RR SEVY1CC1Z T2 Plus CPU module, 8 core, 1.2GHz, 16GB(4x4GB) with memory board RR SEVY1CD1Z T2 Plus CPU module, 8 core, 1.2GHz, 32GB(4x8GB) with memory board SEVY1DD1Z T2 Plus CPU module, 8 core, 1.4GHz, 32GB(4x8GB) with memory board SEVY1DB1Z T2 Plus CPU module, 8 core, 1.4GHz, 8GB(4x2GB) with memory board Subject to 1.4GHz availability SEVY1DC1Z T2 Plus CPU module, 8 core, 1.4GHz, 16GB(4x4GB) with memory board Subject to 1.4GHz availability SESY2A1Z 2GB Memory Expansion kit(2x1GB) SESY2B2Z 4GB Memory Expansion kit(2x2GB) Non-Extended ECC version, no mixing SESY2D1Z 16GB Memory Expansion kit (2 X8 GB) SESY2C1Z 8GB Memory Expansion kit(2x4GB) SESVY3A11Z 73GB 10Krpm, 2.5” SAS Drive SESVY3C11Z 146GB 10Krpm, 2.5” SAS Drive SESVY3A21Z 73GB 15Krpm, 2.5” SAS Drive SEVY9PS51Z 1120W AC Power Supply SEVY9DV1Z DVD, 8x DVD-RW Slimline module SEVY9RK1Z Slide Rail Kit, Screw Mounting SEVY9RK2Z Slide Rail Kit, Toolless Mounting Sun Internal and Approved Partners Only 57
  • 58. Option Parts List for T5440 (cont..) Part Number Option Description Avialable Max numberSG-XPCIE1FC-QF4 Qlogic, 4Gb, PCI-E Single Port RR 8SG-XPCIE2FC-QF4 Qlogic, 4Gb, PCI-E Dual Port RR 8SG-XPCIE1FC-EM4 Emulex, 4Gb, PCI-E Single Port RR 8SG-XPCIE2FC-EM4 Emulex, 4Gb, PCI-E Dual Port RR 8SG-XPCIE2SCSI-U320Z U320 SCSI, PCI-E Dual Port RR 8SG-XPCIE8SAS-E-Z SAS, 8 Port, PCI-E RR 8SGXCPCIESAS-R-EXT-Z SAS, 8 Port External RAID Controller RR 2X7281A-2 Northstar Dual port GigE MMF Low profile PCI-E RR 8X7280A-2 Northstar Dual port GigE UTP Low profile PCI-E RR 8X1236A-Z Infiniband HBA, Dual 4x ports, PCI-E RR 8X4447A-Z Atlas Quad Port, GigE Copper, PCI-E (up to 3/6) RR 8X1027A-Z Atlas Dual Port, 10GigE, PCI-E RR 4X5558A Tranceiver 10GE SR XFP (XAUI/Atlas) RR -X5560A-Z Tranceiver 10GE LR XFP (XAUI/Atlas) RR -X3000A-Z XVR-300, PCI-E x8 Graphics cards (up to 1) RR 4SESY7XA1Z 10GbE XAUI card RR 2TBD 10GbE XAUI Copper 10g Base-T TBD 2X6000A Sun Crypto Accelerator 6000 RR 2SENY8BE1Z External IO Expansion Unit – PCI-E base Config RR 2SENY8TE1Z PCI-Express “boat” for External IO Expansion Unit (6 Hot Plug carriers) RR 4SELY8LK1Z Copper link kit for IO Expansion unit “boat” RR 4 Sun Internal and Approved Partners Only 58
  • 59. Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC VF• Sun SPARC Enterprise T5x40 servers > Overview of Features > Comparison to SunFire T1000/T2000/T5x20 > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40• ILOM 2.0• LDOMs for T5x40• CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ... Sun Internal and Approved Partners Only 59
  • 60. Tuning Out of the Box• Using Solaris 10 05/08 (update 5) with Patches > MPO optimisations used > DMA relaxed Ordering implemented > Default pages sizes for – Initdata Segments is 64kB pages – Private Anon, Heap, Process Stack, Text, Shared Anon segments are 4MB pages – ISM and DISM use 256MB pages – New variables introduced to control these sizes if changes are needed – max_uheap_lpsize,max_ustack_lpsize, max_privmap_lpsize, max_shm_lpsize, max_uidata_lpsize, max_utext_lpsize, max_shm_lpsize > L2 Cache hashing implemented – helps randomise accesses to L2 caches and ease thrashing > Stack biasing implemented – D-Cache is only 4 Way associative... 8 threads – Running multiple copies of a binary can lead process stacks aligning in the cache and to hot lines in the cache – Stack Bias slews the stack variable for processes. Sun Internal and Approved Partners Only 60
  • 61. Tuning Out of the Box (continued..) > Processor Groups introduced – Helps associate CPUs with some shared resources (int/FP pipelines, Crypto units etc) – Enables improved multi-level CMT load balancing and affinity policies > bcopy/uiomove Optimizations – Key kernel operations – new implementation about 2x faster than previous• Optimisations will require patching of S10 05/08 > These will be post RR patches as they become available > Most fixes are already committed in update 5 > Check patch levels for customers with performance issues > Also check Firmware versions as this can improve performance too... Sun Internal and Approved Partners Only 61
  • 62. Tuning Recommendations• Less involved than with T2000• S10 5/08 is more automatic – OOB optimised page size selection, thread scheduling, block copy, IP instances for zones, page coloring etc...• If tuning is necessary remember... > Improved pipeline results in Single threaded performance around 20% better > Floating Point is no longer an issue > Start with T2000 tuning options > Use updated trapstat, cpustat,• LDOMs may be an easier way to scale poorly threaded applications• Solaris Crypto framework is still being tuned > Great for big packets, working on improving small packets• A Blueprint is being prepared (Denis Sheahan) Sun Internal and Approved Partners Only 62
  • 63. Tuning Recommendations (cont..)• Refer to UltraSPARC T1 applications tuning best practices... > http://www.sun.com/servers/coolthreads/tnb/applications.jsp• CoolTools are being updated for UltraSPARC T2 and T2 Plus > http://www.sun.com/servers/coolthreads/overview/cooltools.jsp• Sun Studio 12 is UltraSPARC T1 and T2/T2 Plus aware > Recompiling without code changes should benefit T1/T2/T2 Plus apps• LDOMs configs can benefit significantly from a transition to T5x20 platform > Multiple IO/Service domains with higher bandwidth PCI busses > Make sure appropriate LDOMs tuning and sizing is carried out• Wider Crypto capability, so ensure code is using PKCS#11 framework > Ensure that the ncp (SSL, RSA, DSA etc) and n2cp (bulk ciphers), n2rng(random number generator) drivers are used appropriately Sun Internal and Approved Partners Only 63
  • 64. PCIe Performance• Easily two to four times the performance of T5x20 (two or four PCIe Busses)• All PCI card slots are x8 PCI electrically > Single level hierarchy > Cut through switching > PCI-E relaxed ordering for higher performance• Performance per bus... > Marketing performance = 4000MB/sec > Theoretical Rate with overheads = 3790MB/sec > Max. DMA Achievable(bidirectional) = 3096MB/sec – with 128byte packets – And 512 byte read size Sun Internal and Approved Partners Only 64
  • 65. Agenda• Introduction• Comparision of the UltraSPARC T2 to UltraSPARC VF• Sun SPARC Enterprise T5x40 servers > Overview of Features > System Architecture and Tours > Memory, Networking and IO Expansion > Disks, Fans and Power Supplies > Solaris for T5x40• ILOM 2.0• LDOMs for T5x40• CRUs/FRUs and Configuration Options• Tuning and Performance• Tools, Information and References• Q & A ... Sun Internal and Approved Partners Only 65
  • 66. Tools and Information• Cool Tools – http://www.sun.com/servers/coolthreads/overview/cooltools.jsp• SUNWIN Tokens > Sun SPARC Enterprise T5440 – Architecture Whitepaper : 526122 – JTF : 526118 > T5440 RAS Whitepaper : TBA• Video Tours > Check the Sun.com product pages• Power Calculators > Check the Sun.com product pages Sun Internal and Approved Partners Only 66
  • 67. Further Information• Presto > T5440 > http://sunwebcollab.east.sun.com/gm/document-1.26.680780/Batoka_PRESTO.html• Installed Business Base (IBB) Promotions > http://sunwebcollab.east.sun.com/gm/document-1.26.680782/Batoka_IBB_Promo.html• Start Up Essentials program > http://sunwebcollab.east.sun.com/gm/document-1.26.680794/Maramba_Batoka_SU_April1.html Sun Internal and Approved Partners Only 67
  • 68. Reference Information• T5440 Engineering TOI – http://ittdev.east.sun.com/TechTalk2/EngineeringTOIs/• ILOM/ALOM User Guides – http://www.sun.com/products-n-solutions/hardware/docs/pdf/819-1160-13.pdf – http://www.sun.com/products-n-solutions/hardware/docs/pdf/819-7981-10.pdf• Solaris FMA – http://fma.eng – http://lightside.sfbay/n2-fma/docs.php• Sun Internal and Approved Partners Only 68
  • 69. Thank YouSun Microsystems 69
  • 70. T5x40 PCI Topology Sun Internal and Approved Partners Only 70
  • 71. 10GbE Networking Substantiation10GbE price for 2 x SR-fiber PCI-E adaptors = $6,740. From HP DL380G5 pricing on HP Store, www.hp.com, 12/17/07. Cost for 2 x 10GbEand SR transceivers for Maramba and Batoka are $1,800. From sun.com, 12/17/07 Sun Internal and Approved Partners Only 71