Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

M series technical presentation-part 1

1,377 views

Published on

Download & Share Technology
Presentations http://goo.gl/k80oY0
Student Guide & Best http://goo.gl/6OkI77

Published in: Technology
  • Be the first to comment

M series technical presentation-part 1

  1. 1. 1
  2. 2. Sun SPARC Enterprise Mx000 Servers - A Deep Dive (Part 1) Presenter’s Name Presenter’s Title 8.4
  3. 3. 3 Sun SPARC Enterprise Midrange and High-end Servers
  4. 4. 4 • Mainframe-class reliability • Highly scalable performance • Advanced virtualization • Seamless scalability • Easy upgradability • Balanced SMP design • Price/performance targeted • Competitive RAS and S/W • Dense rack mount package • Highest availability • Highest absolute performance • Sophisticated resource control • Highest scalability • Entry level enterprise system • Mainframe-class RAS • Dense rack mount package SPARC Enterprise Servers M4000 and M5000 M8000 and M9000 M3000
  5. 5. 5 Term Chart Term Definition CPU memory board Unit (CMU) Unit equipped with a CPU module and memory IO board unit (IOU) Unit equipped with multiple PCI cards and disk drives (M8000/M9000 only) eXtended System Board (XSB) Either the entire CMU (Uni-XSB) or a quarter of the a CMU (Quad- XSB). Uni-XSB sometimes referred to as just PSB. Quad-XSB sometimes referred to as XSB. Logical System Board (LSB) A number assigned to an XSB to identify it in a domain XBU (M9000 Only) Cross Bar Unit. Connects CMUs & IOUs CLKU (M9000 Only) Clock Unit that provides Clock for ASICs. BPA (M8000 Only) Backplane with integrated XB and CLK XSCFU eXtended System Control Facility. Three types: XSCFU (M3000- M5000), XSCFU_B (M8000, M9000-32, Base cab of M9000-64), and XSCFU_C (Exp cab of M9000-64)
  6. 6. 6 <Insert Picture Here> Entry Level Systems – M3000
  7. 7. 7 Updated Entry Level Enterprise Server • 2U Rackmount • Single SPARC64 VII quad-core processor – 2.75GHz/5MB$, four core with two SMT threads per core – Mainframe-heritage RAS features • Up to 64GB main memory, with 8GB DIMM • Four PCI-Express (low-profile) slots •Four disk bays (2.5” SAS) •On-board four Gigabit Ethernet, one SAS port (two wide) •XSCF; eXtended System Control Facility – XCP1091 •Solaris 10 10/08 (U6) + MU8 Sun SPARC Enterprise M3000 Server
  8. 8. 8 Data Centric Network Centric Multi-Thread Performance Single-Thread Performance SPARC Enterprise M3000 server SPARC Enterprise T5220 server Sun Fire V490 server M3000 Workloads The SPARC Enterprise M3000 is suitable for a broad application space; batch/DB as well as AP/network front-end server SPARC64TM VII Provides Balanced, High-Performance
  9. 9. 9 Product Comparison M3000 T5220 V490 Processor SPARC64 VII 2.75 GHz, 5 MB L2$ Max 1 (4 cores) UltraSPARC T2 1.4/1.6 GHz, 4 MB L2$ Max 1 (4,6, or 8 cores) UltraSPARC IV+ 2.1 GHz, 2 MB L2$, 32 MB L3$ Max 4 (8 cores) Memory Max 64 GB (8 DIMMs) Max 128 GB (16 DIMMs) Max 64 (16 DIMMs) Internal Disks Max 4 (2.5” SAS) Max 16 (2.5” SAS) Max 2 (2x FC-AL) Removable Media DVD DVD DVD On-Board I/O 4 GbE, 2 10 GbE, 1 x2 SAS, 1 Serial, 1 USB, 2 UPC 4 GbE, 1 Serial, 4 USB 2 GbE, 1FC, 3 Serial, 2 USB PCI Slots 4 PCIe 6 PCIe 6 PCI Enclosure 2U 2U 5U Redundant Parts Disk, PSU, Fan Disk, PSU, Fan Disk, PSU, Fan Hot-Swappable Parts Disk, PSU, Fan Disk, PSU, Fan Disk, PSU, Fan Power Options 1-phase, 1+1 cables 1-phase, 1+1 cables 1-phase, 1+1 cables Power Consumption Max 400W Max 896W Max 1,750W System Management XSCF ILOM RSC
  10. 10. 10 Hardware Specifications Measurement North America Metric 2RU 2RU Height 3.43 in 8.7 cm Width 17.4 in 44 cm Depth 25.91 in 65.7 cm Weight (max.) 50.6 lb. 22 kg
  11. 11. 11 Block Diagram Slot 0 Slot 1 Slot 2 Slot 3
  12. 12. 12 M3000 IOU Device Paths
  13. 13. 13 I/O Slot to Device Path Tables IOU Slot Slot Type XSB Device Path Slot 0 PCIe 00-0 /pci@0,600000/pci@0/pci@8 Slot 1 PCIe 00-0 /pci@1,700000/pci@0/pci@0 Slot 2 PCIe 00-0 /pci@1,700000/pci@0/pci@8 Slot 3 PCIe 00-0 /pci@1,700000/pci@0/pci@9 Accessible Internal Devices Physical Location XSB Device Path Network Port 0 System 00-0 /pci@0,600000/pci@0/pci@1/pci@0/network@4 Network Port 1 System 00-0 /pci@0,600000/pci@0/pci@1/pci@0/network@4,1 Network Port 2 System 00-0 /pci@0,600000/pci@0/pci@2/pci@0/network@4 Network Port 4 System 00-0 /pci@0,600000/pci@0/pci@2/pci@0/network@4,1 HD0 System 00-0 /pci@0,600000/pci@0/pci@0/scsi@0/disk@0 HD1 System 00-0 /pci@0,600000/pci@0/pci@0/scsi@0/disk@1 HD2 System 00-0 /pci@0,600000/pci@0/pci@0/scsi@0/disk@2 HD3 System 00-0 /pci@0,600000/pci@0/pci@0/scsi@0/disk@3 DVD System 00-0 /pci@0,600000/pci@0/pci@0/scsi@0/disk@4 SAS Port System 00-0 /pci@0,600000/pci@0/pci@0/scsi@0/disk@5 or /pci@0,600000/pci@0/pci@0/scsi@0/disk@6
  14. 14. 14 System Cabinet Hard Disk DrivesPower Switch DVD DriveStatus Indicators (Power, XSCF ready, System check) Key Switch (Lock, Service) PCI slots Gigabit Ethernet UPC RCI SAS Front View Rear View USB Serial LAN 0 & 1
  15. 15. 15 DIMMSystem controller Processor ModuleMCBI-SCF RAM for firmware XSCF Processor XSCF DVD connector Switch FLP I/O controller (Oberon) DCDC converter Temperature sensor Detailed View
  16. 16. 16 Key Features (cont.) • Eco-friendly • Small size; system size is almost same as T5220 • Low power; Max. Watts less than T5220/T5120 • Multi-speed Fans provide optimized noise according to temperature; operating acoustic noise ≤ 50dBA(target) • Improved System Reliability/Availability • Data validation by ECC/parity mechanism btw major components • Dual Power Feed, as well as hot-swappable/redundant Fan, PSU and Disk • Online firmware upgrade • Harmonized manageability with M-Series Servers • XSCF provides same user interface and remote notification
  17. 17. 17 JSC: Jupiter System Controller • JSC is a chip, which is combined functionalities of SC and MAC chips used in M4000-M9000 systems. • Control the mutual access among CPU, memory and IO • Features: • Consists of 1 JSC chip, and has the interface to 1 CPU and 1 Oberon. • Internal 1224MHz operation • Connects with CPU at 1224MHz • Contains 612 MHz DDR-II interface and supports up to 8GB for DIMM capacity.
  18. 18. 18 PCI-Express Sub-System •PCIe (Gen 1) x8 lane support •Not hot-pluggable •No PCI-X support •Low profile only 4 x GbE PCIe Bridge “Oberon” PCIe Switch PCIexpress Slot PCIexpress Slot PCIexpress Slot PCIexpress Slot 8 8 8 8 G-Ether SAS 4 Disks DVD External SAS Interface (2 wide) PCIe Switch
  19. 19. 19 Cooling •2 cooling zones •3 variable speed fans Zone 1 Zone 2 = Sensor(intake) = Sensor (CPU) = Sensor (exhaust)
  20. 20. 20 Power Condition: Single Phase AC • Rated voltage: AC100-120 / 200-240 V +/- 10% • Frequency: 50/60Hz (+2% / -4%) • Power consumption: 505W max • AC plug: 2 plugs • 100V & 200V • IEC60309 16A 250V (RoW w/o Japan,Korea and Taiwan) • IEC60320-C14/13 250V 15A(NA,Japan) • 2 PSU standard
  21. 21. 21 A word on power consumption.... • M3000 is listed as 505W max • Actual power draw based on max config is 400W • Quote both actual lines draw and the power supply rating
  22. 22. 22 Managed FAN Speed Room temperature affects sound pressure level Sound pressure level of server at Celsius 25 is the same as the quiet office, 47dB * Sun Fire V445 rates 77dB; SPARC Enterprise T5220 rates 74dB Level1 (~ 19.9) Level2 (19.0~ 21.9) Level3 (21.0~ 23.9) Level4 (23.0~ 25.9) Level5 (25.0~ 27.9) Level6 (27.0~ 29.9) Level7 (29.0~ 31.9) Level8 (31.0~ 33.9)Level9 (33.0~ ) 47 dB 25 degrees C Sound Pressure [dB] Rang of Temperature [Celsius] 42 46 50 Sound Pressure [dB]
  23. 23. 23 MBU_B • Main board, integrated MBU, CPUM, MEMB, IOU, and DDCR which existed on M4000/M5000 system • 4 core CPU is soldered. In case of broken, MBU_B itself will be replaced • Eight DDR2 (612 MHz) SDRAM slots • 2-way interleave • Four low profile PCIe slots
  24. 24. 24 MBCI-SCF • MBCI - Maintenance Bus Controller Interface • MBCI-SCF is a communication device between MBU and XSCF • Access to POST/OBP FMEM • Emulation of serial port • Communication interface to XSCF • Provides maintenance bus MBU XSCF CPU JSC FMEM (POST/OBP) MBCI-SCF SRAM PPC
  25. 25. 25 Operator Panel • Same specifications as current M-Series systems • 3 LEDS, 1 push switch, 1 key switch • System Power LED, XSCF ready LED and System check LED • System power on/off; Push switch • Key switch; Lock or Service
  26. 26. 26 Memory Installation • There are 8 DIMM slots on an MBU Group-A: x4 slots Group-B: x4 slots • Install Group-A, first • Install same type of DIMMs to slots in a group • Same type means same capacity and same rank • Capacity of DIMM in Group-B must be smaller or equal to DIMM in Group-A Ex.1: Group-A: 2GB 1rank DIMM Group-B: 1GB 1rank DIMM Ex.2: Group-A: 2GB 1rank DIMM Group-B: 2GB 1rank DIMM
  27. 27. 27 System Bandwidth M3000 (2.52 GHz) M3000 (2.75 GHz) Stream (Copy) 5.6 GB/s 6.4 GB/s Stream (Triad) 4.5 GB/s 5.1 GB/s Peak System Bandwidth a 17 GB/s 20 GB/s Peak I/O Bandwidth b 4 GB/s 4 GB/s a. Theoretical system bandwidth at peak is calculated by multiplying the bus width by the bus frequency between the SC and the MAC. b. Theoretical I/O bandwidth at peak is calculated by multiplying the bus width by the bus frequency between the SC and the PCIe bridge.
  28. 28. 28 Degradation (Page) Dual power feed Reliability at Component Level Memory CPU Dynamic Degradation (Cache Way or Core) Fans Redundancy Hot-swap Hardware Power Supply Units Redundancy Hot-swap Disk Drives Redundanc y Hot-swap * Available with Software Minimizes Unplanned Downtime Minimizes Planned Downtime
  29. 29. 29 M3000% prtdiag System Configuration: Sun Microsystems sun4u Fujitsu SPARC Enterprise M3000 Server System clock frequency: 1224 MHz Memory size: 65536 Megabytes ==================================== CPUs ==================================== CPU CPU Run L2$ CPU CPU LSB Chip ID MHz MB Impl. Mask --- ---- -------------------------------------- ---- --- ----- ---- 00 0 0, 1, 2, 3, 4, 5, 6, 7 2750 5.0 7 160 ============================ Memory Configuration ============================ Memory Available Memory DIMM # of Mirror Interleave LSB Group Size Status Size DIMMs Mode Factor --- ------ ------------------ ------- ------ ----- ------- ---------- 00 A 32768MB okay 8192MB 4 no 2-way 00 B 32768MB okay 8192MB 4 no 2-way ========================= IO Cards ========================= LSB Name Model --- --------- ------------ 00 scsi LSI,1068E 00 network N/A 00 network N/A 00 network N/A 00 network N/A ==================== Hardware Revisions ==================== System PROM revisions: ---------------------- OBP 4.24.11 2009/04/21 14:53 =================== Environmental Status =================== Mode switch is in UNLOCK mode =================== System Processor Mode =================== SPARC64-VII mode System Details
  30. 30. 30 Differences With Current M-Series Servers • Chips for on-board devices • M4000-M9000: SAS1064 X 1, BCM5704C X 1, PEX8532 X 1 • M3000: SAS1068E X 1, BCM5715C X 2, PEX8533 X 2 • External SAS i/f: newly added • New ASIC JSC: SC + MAC • Single mother board structure: MBU • M4000/M5000: MBU+CPUM+MEMB+IOU+DDCR • Reduction in boards provides higher reliability • Change of power/cooling subsystem • Number of FAN/DDC are changed
  31. 31. 31 Differences With Current M-Series Servers • SC to MAC data path • M4000-M9000 has ECC on data path • JSC uses parity for SC-MAC communication • No support for the I/O Expansion Unit • Domaining (These details will be covered later) • Single domain. ID is 0 • LSB=0 • SB0 is in uni-xsb mode • No DR support • No memory mirroring
  32. 32. 32 External SAS Device Support
  33. 33. 33 Introduction to SAS • Serial Attached SCSI • Point to point serial communication protocol to transfer data to/from devices • SAS specification defines 3 protocols • SSP – Serial SCSI Protocol • STP – SATA Tunneling Protocol • SMP – SAS Management Protocol
  34. 34. 34 Storage Protocol Comparison
  35. 35. 35 Advantages of SAS • Advantages over SATA: • Multi host support • More robust error recovery and reporting • Encapsulates SATA protocol • Advantages over parallel SCSI: • Supports larger number of devices (over 16K) • Limited bus contention and full speed realized from Initiator/Target • Does not require termination • Advantages over FiberChannel: • Cheaper • Less complex • Supports SATA drives
  36. 36. 36 SAS – Connector Icons • SAS defines 3 receptacle icons • Circle – enclosure in • Diamond – enclosure out • Circle and Diamond – enclosure universal (Sun default) • SAS also defines connector keys that prevent incorrect cabling
  37. 37. 37 Understanding SAS “Wide” Ports • Scaleable connect • “x4 wide” creates 4 lanes in a single connector • A single SAS lane is 3 Gb/s • The M3000 uses 2 wide, or 2 lanes • M3000 SAS port can drive 6Gb/s • Can support 117 external devices using built-in SAS connection
  38. 38. 38 <Insert Picture Here> Mid-Range Systems – M4000 and M5000
  39. 39. 39 M4000/M5000: Mid-Range Systems Overview • Application acceleration • Uses dual core SPARC64 VI and quad core SPARC64 VII processors • Can mix processors in same system and domains • Less down time • Greatly improved RAS functionality • Economical, modular datacenter design • 6U and 10U, rack-optimized enclosures • Extended domaining and dynamic reconfiguration support • Up to 4 fully usable domains • Solaris 10 operating system
  40. 40. 40 Side note: Getting Our Terms Straight Uni-XSB: When the system board is not logically divided, it consists of a single XSB Quad-XSB: When the system board is logically divided, it consists of four XSBs
  41. 41. 41 Mid-range Product Data Sheet M4000 M5000 Processor SPARC64 VI 2.15 GHz, 5 MB L2$ Max 4 (8 cores) SPARC64 VII 2.4 GHz, 5 MB L2$ 2.53 GHz, 5.5 MB L2$ Max 4 (16 cores) SPARC64 VI 2.15 GHz, 5 MB L2$ Max 8 (16 cores) SPARC64 VII 2.4 GHz, 5 MB L2$ 2.53 GHz, 5.5 MB L2$ Max 8 (32 cores) SPARC64 VII+ 2.66 GHz, 11 MB L2$ Max 8 (32 cores) SPARC64 VII+ 2.66 GHz, 11 MB L2$ Max 4 (16 cores) Memory Max 256 GB (32 DIMMs) Max 512 GB (64 DIMMs) Internal Disks Max 2 (2.5” SAS) Max 4 (2.5” SAS) Removable Media DVD, DAT On-Board I/O 2 GbE, 2 10/100E,1 Serial, 1 USB, 2 UPC 4 GbE, 2 10/100E, 1 Serial, 1 USB, 2 UPC PCI Slots 4 PCIe, 1 PCI-X 8 PCIe, 2 PCI-X Enclosure 6U 10U Redundant Parts Disk, PSU, Fan Hot-Swappable Parts Disk, PSU, Fan Power Options 1-phase, 1+1 cables 1-phase, 2+2 cables Power Consumption Max 1390W Max 2677W System Management XSCF (service processor) Domains Max 2 Max 4
  42. 42. 42 Mid-range Product Comparison M4000 V490 Processor SPARC64 VI 2.15 GHz, 5 MB L2$ Max 4 (8 cores) SPARC64 VII 2.4 GHz, 5 MB L2$ 2.53 GHz, 5.5 MB L2$ Max 4 (16 cores) UltraSPARC IV+ 2.1 GHz, 2 MB L2$, 32 MB L3$ Max 4 (8 cores) SPARC64 VII+ 2.66 GHz, 11 MB L2$ Max 4 (16 cores) Memory Max 256 GB (32 DIMMs) Max 64 (16 DIMMs) Internal Disks Max 2 (2.5” SAS) Max 2 (2x FC-AL) Removable Media DVD, DAT DVD On-Board I/O 2 GbE, 2 10/100E,1 Serial, 1 USB, 2 UPC 2 GbE, 1FC, 3 Serial, 2 USB PCI Slots 4 PCIe, 1 PCI-X 6 PCI Enclosure 6U 5U Redundant Parts Disk, PSU, Fan Disk, PSU, Fan Hot-Swappable Parts Disk, PSU, Fan Disk, PSU, Fan Power Options 1-phase, 1+1 cables 1-phase, 1+1 cables Power Consumption Max 1390W Max 1,750W System Management XSCF (service processor) RSC Domains Max 2 Max 1
  43. 43. 43 Mid-range Product Comparison M5000 V890 Processor SPARC64 VI 2.15 GHz, 5 MB L2$ Max 8 (16 cores) SPARC64 VII 2.4 GHz, 5 MB L2$ 2.53 GHz, 5.5 MB L2$ Max 8 (32 cores) UltraSPARC IV+ 2.1 GHz, 2 MB L2$, 32 MB L3$ Max 8 (16 cores) SPARC64 VII+ 2.66 GHz, 11 MB L2$ Max 8 (32 cores) Memory Max 512 GB (64 DIMMs) Max 128 (32 DIMMs) Internal Disks Max 4 (2.5” SAS) Max 12 Removable Media DVD, DAT DVD, DDS-4 On-Board I/O 4 GbE, 2 10/100E, 1 Serial, 1 USB, 2 UPC 1 GbE PCI Slots 8 PCIe, 2 PCI-X 9 PCI Enclosure 10U 17.5U Redundant Parts Disk, PSU, Fan Disk, PSU, Fan Hot-Swappable Parts Disk, PSU, Fan Disk, PSU, Fan Power Options 1-phase, 2+2 cables 1-phase, 2+1 cables System Management XSCF (service processor) RSC Domains Max 4 Max 1 Peak Bandwidth 64 GB/s 9.6 GB/s I/O Bandwidth 16 GB/s 1.2 GB/s
  44. 44. 44 Methodology Theoretical Peak Data B/W • The data B/W values come from the SC (System Controller) to MAC (Memory Access Controller) bus- width multiplied by the frequency, then multiplied by the number of SB (System Boards). M4000: 32GB/s = (16B(in)+16B(out)) x 0.506GHz x 2SC x 1SB M5000: 64GB/s = (16B(in)+16B(out)) x 0.506GHz x 2SC x 2SB
  45. 45. 45 Methodology Theoretical Peak I/O B/W • The I/O B/W values come from the SC (System Controller) to Oberon (PCI Host Bridge) bus-width multiplied by the frequency. M4000: 8GB/s = (4B(in)+4B(out)) x 0.506GHz x 2SC x 1IOB M5000: 16GB/s = (4B(in)+4B(out)) x 0.506GHz x 2SC x 2IOB
  46. 46. 46 Mid-range System Servers M4000 System M5000 System Front Front Back Back Disk PSU DVD CPU Memory DAT Fan PCI XSCF CPU Fan Disk DAT PSU DVD Memory XSCF PCI
  47. 47. 47 M4000 Product Concept Front Iso View Rear Iso View 2 x 172mm Fans 2 x PSU 2 x HDD, DVD, DAT 4 x MEM Modules 1 x IO Tray (1 PCI-X, 4 PCI-Express) XSCFU 2 x CPU Modules
  48. 48. 48 M5000 Product Concept Front Iso View Rear Iso View 4 x 172mm Fans 4 x PSU 4 x HDD, DVD, DAT 8 x MEM Modules 4 x CPU Modules 2 x IO Trays (1 PCI-X, 4 PCI-Express per Tray) XSCFU
  49. 49. 49 Mid-Range Systems Components M4000 Motherboard CPU Module Connectors Memory Board Connectors SC ASICs
  50. 50. 50 Mid-Range Systems Components CPU Module •2.15GHz SPARC64 VI •2.4GHz/2.53GHz SPARC64 VII •2.66GHz SPARC64 VII+ •2 CPUs per module •Both CPUs the same type and speed •2 (M4000) or 4 (M5000) modules per system
  51. 51. 51 Mid-Range Systems Components Memory Board •8 DIMM slots •1, 2, 4, and 8GB DIMMs •ECC •Extended-ECC •Mirrored •4 (M4000) or 8 (M5000) modules per system •Minimum is 4 x 1GB DIMMs
  52. 52. 52 Mid-Range Servers I/O Tray Components •Controls DVD, DAT •Controls 2 SAS disks •4 x PCIe •1 x PCI-X •2 GbE ports •1 tray in M4000 •2 trays in M5000 I/O Tray
  53. 53. 53 System Views: M4000
  54. 54. 54 System Views: M5000
  55. 55. 55 PCI EXPRESS (switched point to point) PCI vs PCIexpress I/O Technology PCI (shared bus)
  56. 56. 56 PCIe Bandwidth Technology Bus Speed Bus Width Bandwidth Comments PCI Shared Bus 33 MHz 32 bit 133 MB/s 33 MHz 64 bit 266 MB/s 66 MHz 32 bit 266 MB/s 66 MHz 64 bit 532 MB/s PCI-X Shared Bus 66 MHz 64 bit 512 MB/s 100 MHz 64 bit 800 MB/s 133 MHz 64 bit 1 GB/s Current supported speed 266 MHz 64 bit 2 GB/s PCI Express Switched Point to Point x1 500 MB/s x2 1 GB/s x4 2 GB/s x8 4 GB/s Current supported speed x12 6 GB/s x16 8 GB/s x32 16 GB/s
  57. 57. 57 PCICS: PCI Cassette for M4000/M5000 IO Cage In PCICS, PCIexpress Short dedicated cassette provided Carrier main body (metal) Moveable carrier plate (plastic) Carrier slot keyholes Operating handle Card retainer (5 total, 3 types)
  58. 58. 58 2 x GbE 4 PCIe to PCI-X Bridge PCI-X Slot IOU PCIe Bridge PCIe Switch PCIe Switch PCIe Switch 8PCIexpress Slot PCIexpress Slot PCIexpress Slot PCIexpress Slot 8 8 8 8 CPU Module CPU Module Internal Storage DVD SC1SC0 System Control Facility SATA x4 8 DIMMs MAC1 8 DIMMs MAC0 CPU0 CPU1 Memory Board Memory Board CPU2 CPU3 MAC2 Memory Board MAC3 8 DIMMs 8 DIMMs Memory Board Midrange M4000 Block Diagram
  59. 59. 59 SC ASIC • The SC (System Controller) ASIC is responsible for setup and enforcement of domaining. • The multiple paths are for performance, not redundancy • The physical address space on each SB or CMU is divided equally across the SCs • Each SC stores L2$ tags for the processors on the same SB or CMU.
  60. 60. 60 Midrange M5000 Block Diagram CPU Module CPU Module 8 DIMMs MAC1 8 DIMMs MAC0 CPU0 CPU1 Memory Board Memory Board CPU2 CPU3 MAC2 Memory Board MAC3 8 DIMMs 8 DIMMs Memory Board IOU #0 and Disks 0,1 and DVD/DAT CPU Module CPU Module 8 DIMMs MAC4 8 DIMMs MAC5 CPU4 CPU5 Memory Board Memory Board CPU6 CPU7 MAC 6 Memory Board MAC7 8 DIMMs 8 DIMMs Memory Board IOU #1 and Disks 2,3 SC3SC2 SC1SC0
  61. 61. 61 CPU Module CPU Module MAC1 CPU0 CPU1 Memory Board Memory Board CPU2 CPU3 Memory Board Memory Board Midrange M4000 Mirrored Memory = Group A = Group B 4 DIMMs Mirro r 4 DIMMs MAC0 MAC2 MAC3 4 DIMMs Mirro r 4 DIMMs 4 DIMMs Mirro r 4 DIMMs 4 DIMMs Mirro r 4 DIMMs
  62. 62. 62 CPU Module CPU Module MAC1 CPU0 CPU1 Memory Board Memory Board CPU2 CPU3 Memory Board Memory Board Midrange M5000 Mirrored Memory = Group A = Group B 4 DIMMs Mirro r 4 DIMMs MAC0 MAC2 MAC3 4 DIMMs Mirro r 4 DIMMs 4 DIMMs Mirro r 4 DIMMs 4 DIMMs Mirro r 4 DIMMs CPU Module CPU Module MAC5 CPU4 CPU5 Memory Board Memory Board CPU6 CPU7 Memory Board Memory Board 4 DIMMs Mirro r 4 DIMMs MAC4 MAC6 MAC7 4 DIMMs Mirro r 4 DIMMs 4 DIMMs Mirro r 4 DIMMs 4 DIMMs Mirro r 4 DIMMs
  63. 63. 63 Memory Mirroring Systems with no memory mirroring Memory mirroring on Mx000 Error Error Memory Memory 0 Memory 1 Memory 0 Memory 1 Memory Controller Memory Controller Memory Controller Data read from memory Data read from memoryData write to memory  On write, same data is written to both memory 0 & 1  On reads, reads from memory 0 & 1 Even with uncorrectable error in memory 0, correct data in memory 1 is used and business continuity is secured.
  64. 64. 64 Memory Memory Patrolling Memory Patrol (other vendors) Mx000 Servers Memory Patrol Detection of memory corruption now earlier CPU Memory Controller OS Memory Memory Controller CPU OSSoftware Memory Patrolling HW (MAC) incorporates memory patrolling function Burden on CPU Swift patrolling not enabled No burden on CPU Swift patrolling enabled Memory patrolling may affect application performance, so the frequency of patrolling is limited Less influence on applications, timely patrolling enabled
  65. 65. 65 Mid-Range M5000 I/O Block Diagram M5000 Main Board 2 x GbE 4 PCIe to PCI-X Bridge PCI-X Slot PCIe Bridge PCIe Switch PCIe Switch PCIe Switch 8 PCIexpress Slot PCIexpress Slot PCIexpress Slot PCIexpress Slot 8 8 8 8 2 x GbE 4 PCIe to PCI-X Bridge PCI-X Slot IOU #0 PCIe Bridge PCIe Switch PCIe Switch PCIe Switch 8 PCIexpress Slot PCIexpress Slot PCIexpress Slot PCIexpress Slot 8 8 8 8 IOU #1
  66. 66. 66 Detailed M5000 Block Diagram
  67. 67. 67 Extended System Board on M4000 M4000 IOUM4000 MBU MEM Board MEM Board MEM Board MEM Board PCIe Switch CPU Module Uni-XSB 00-0 DVD/D AT Disks #0 & #1CPU CPU CPU Module CPU CPU PCI-X Slot PCIe Slot 0 PCIe Slot 1 PCIe Slot 2 PCIe Slot 3 PCIe Switch
  68. 68. 68 Extended System Board on M4000 M4000 IOUM4000 MBU MEM Board MEM Board MEM Board MEM Board PCIe Switch CPU Module Quad-XSB 00-3 Quad-XSB 00-2 Quad-XSB 00-1 Quad-XSB 00-0 DVD/D AT Disks #0 & #1CPU CPU CPU Module CPU CPU PCI-X Slot PCIe Slot 0 PCIe Slot 1 PCIe Slot 2 PCIe Slot 3 PCIe Switch
  69. 69. 69 Extended System Board on M5000 M5000 IOU #1M5000 MBU MEM Board MEM Board MEM Board MEM Board Uni-XSB 00-0 MEM Board MEM Board MEM Board MEM Board Uni-XSB 01-0 CPU Module CPU CPU CPU Module CPU CPU CPU Module CPU CPU CPU Module CPU CPU SB0 SB1 M5000 IOU #0PCI-X Slot PCIe Slot 2 PCIe Slot 3 PCIe Slot 0 PCIe Slot 1 PCI-X Slot PCIe Slot 2 PCIe Slot 3 PCIe Slot 0 PCIe Slot 1 DVD/D AT Disk 0,1 Disk 2,3 IO Ctl r IO Ctl r IO Ctl r IO Ctl r
  70. 70. 70 Extended System Board on M5000 M5000 MBU MEM Board MEM Board MEM Board MEM Board Quad-XSB 00-3 Quad-XSB 00-2 Quad-XSB 00-1 Quad-XSB 00-0 MEM Board MEM Board MEM Board MEM Board Quad-XSB 01-3 Quad-XSB 01-2 Quad-XSB 01-1 Quad-XSB 01-0 CPU Module CPU CPU CPU Module CPU CPU CPU Module CPU CPU CPU Module CPU CPU SB0 SB1 M5000 IOU #1 M5000 IOU #0 PCI-X Slot PCIe Slot 2 PCIe Slot 3 PCIe Slot 0 PCIe Slot 1 PCI-X Slot PCIe Slot 2 PCIe Slot 3 PCIe Slot 0 PCIe Slot 1 IO Ctl r IO Ctl r IO Ctl r IO Ctl r DVD/D AT Disk 0,1 Disk 2,3
  71. 71. 71 Uni-XSB 01-0 Extended System Board on M5000 M5000 MBU MEM Board MEM Board MEM Board MEM Board Quad-XSB 00-3 Quad-XSB 00-2 Quad-XSB 00-1 Quad-XSB 00-0 MEM Board MEM Board MEM Board MEM Board CPU Module CPU CPU CPU Module CPU CPU CPU Module CPU CPU CPU Module CPU CPU SB0 SB1 M5000 IOU #1 M5000 IOU #0 PCI-X Slot PCIe Slot 2 PCIe Slot 3 PCIe Slot 0 PCIe Slot 1 PCI-X Slot PCIe Slot 2 PCIe Slot 3 PCIe Slot 0 PCIe Slot 1 IO Ctlr IO Ctlr IO Ctlr IO Ctlr DVD/D AT Disk 0,1 Disk 2,3
  72. 72. 72 Valid Memory Board Configurations Per System Board Uni-XSB Mode Quad-XSB Mode = Memory Module= CPU Board
  73. 73. 73 M4000/M5000 Power Cords = C19 to C20 Jumper cable ships standard with each M4000/M5000 •Recommended rack is Sun Rack 1000-42 with the Modular Power System (MPS) •With Sun Rack 1000-42, using MPS, the power strips with C13 receptacles must be unplugged from the MPS •Standard jumper is 1.5m. Order SELX9P31z for the 2.5m cable.
  74. 74. 74 <Insert Picture Here> High-End Systems – M8000, M9000
  75. 75. 75 M8000/M9000: High-End Systems Overview • Application acceleration • Uses dual core SPARC64 VI and quad core SPARC64 VII processors • High performance interconnect • Industry standard PCIe I/O • State-of-the-Art RAS capabilities • Unmatched configuration flexibility • Modular building block architecture (CMU, IOU, expansion box, XSCF) • Initial SMP architecture up to 256 cores • Expanded dynamic system domains • Up to 24 fully usable domains • Fine granularity of unit of domain 1 CPU chip, 4 DIMMs, 2 PCIe slots
  76. 76. 76 High-end Product Data Sheet M8000 M9000-32 M9000-64 Processor SPARC64 VI & VII VI 2.28/2.4 GHz, 5 MB L2$ VII: 2.52/2.88 GHz, 6 MB L2$ Max 16 (64 cores) SPARC64 VI & VII VI 2.28/2.4 GHz, 5 MB L2$ VII: 2.52/2.88 GHz, 6 MB L2$ Max 32 (128 cores) SPARC64 VI & VII VI 2.28/2.4 GHz, 5 MB L2$ VII: 2.52/2.88 GHz, 6 MB L2$ Max 64 (256 cores) SPARC64 VII+ 3.0 GHz,12 MB L2$ Max 16 (64 cores) SPARC64 VII+ 3.0 GHz,12 MB L2$ Max 32 (128 cores) SPARC64 VII+ 3.0 GHz,12 MB L2$ Max 64 (256 cores) Memory Max 1 TB (128 DIMMs) Max 2 TB (256 DIMMs) Max 4 TB (512 DIMMs) Internal Disks Max 16 (2.5” SAS) Max 32 (2.5” SAS) Max 64 (2.5” SAS) Removable Media DVD, DAT CMU and IOU Max 4 each Max 8 each Max 16 each PCI Slots Max 32 PCIe Max 64 PCIe Max 128 PCIe Domains Max 16 Max 24 Redundant Parts CMU, IOU, XSCF, Disk, PSU, Fans in Fan Tray, XBU CMU, IOU, XSCF, Disk, PSU, Fans in Fan Tray, XBU, CLKU Hot-Swappable Parts CMU, IOU, XSCF, Disk, PSU, Fan Tray Power Options 1-phase/3-phase/Dual Grid Power Consumption Max 7.5KW Max 14.6KW Max 29.9KW System Management XSCF (service processor)
  77. 77. 77 High-end Product Comparison M8000 E6900 Processor SPARC64 VI & VII VI 2.28/2.4 GHz, 5 MB L2$ VII: 2.52/2.88 GHz, 6 MB L2$ Max 16 (64 cores) UltraSPARC IV+ 1.95 GHz, 2 MB L2$, 32 MB L3$ Max 24 (48 cores) SPARC64 VII+ 3.0 GHz,12 MB L2$ Max 16 (64 cores) Memory Max 1 TB (128 DIMMs) Max 384 GB (192 DIMMs) Internal Disks Max 16 (2.5” SAS) None Removable Media DVD, DAT None CMU and IOU Max 4 each 6 Uniboards, 4 I/O Boats PCI Slots Max 32 PCIe Max 24 PCI-X/100, 8 PCI Domains Max 16 Max 4 Redundant Parts CMU, IOU, XSCF, Disk, PSU, Fans in Fan Tray, XBU Uni-Bd, IO-Bd, SC, PSU, Fans, FireplaneSwitch Hot-Swappable Parts CMU, IOU, XSCF, Disk, PSU, Fan Tray Uni-Bd, IO-Bd, SC, PSU, Fans Power Options 1-phase/3-phase/Dual Grid 1-phase/Dual Grid System Management XSCF (service processor) SC (System Controller) System Bandwidth 184 GB/s 9.6 GB/s I/O Bandwidth 61 GB/s 8 GB/s
  78. 78. 78 High-end Product Comparison M9000 E25K Processor SPARC64 VI & VII VI 2.28/2.4 GHz, 5 MB L2$ VII: 2.52/2.88 GHz, 6 MB L2$ Max 64 (256 cores) UltraSPARC IV+ 1.8 GHz, 2 MB L2$, 32 MB L3$ Max 72 (144 cores) SPARC64 VII+ 3.0 GHz,12 MB L2$ Max 64 (256 cores) Memory Max 4 TB (512 DIMMs) Max 1 TB (576 DIMMs) Internal Disks Max 64 (2.5” SAS) None Removable Media DVD, DAT None CMU and IOU Max 16 each Max 18 each PCI Slots Max 128 PCIe Max 72 PCI-X Domains Max 24 Max 18 Redundant Parts CMU, IOU, XSCF, Disk, PSU, Fans in Fan Tray, XBU, CLKU Uni-Bd, IO-Bd, SC, PSU, Fans, Backplane Hot-Swappable Parts CMU, IOU, XSCF, Disk, PSU, Fan Tray Uni-Bd, IO-Bd, SC, PSU, Fans Power Options 1-phase/3-phase/Dual Grid 1-phase/Dual Grid System Management XSCF (service processor) SC (System Controller) System Bandwidth 737 GB/s 43 GB/s I/O Bandwidth 244 GB/s 40 GB/s
  79. 79. 79 Methodology Theoretical Peak Data B/W • The data B/W values come from the SC (System Controller) to MAC (Memory Access Controller) bus- width multiplied by the frequency, then multiplied by the number of SB (System Boards). M4000: 32GB/s = (16B(in)+16B(out)) x 0.506GHz x 2SC x 1SB M5000: 64GB/s = (16B(in)+16B(out)) x 0.506GHz x 2SC x 2SB M8000: 184GB/s = (16B(in)+8B(out)) x 0.480GHz x 4SC x 4SB M9000 (32 CPUs): 368GB/s = (16B(in)+8B(out)) x 0.480GHz x 4SC x 8SB M9000 (64 CPUs): 737GB/s = (16B(in)+8B(out)) x 0.480GHz x 4SC x 16SB
  80. 80. 80 Methodology Theoretical Peak I/O B/W • The I/O B/W values come from the SC (System Controller) to Oberon (PCI Host Bridge) bus-width multiplied by the frequency. M4000: 8GB/s = (4B(in)+4B(out)) x 0.506GHz x 2SC x 1IOB M5000: 16GB/s = (4B(in)+4B(out)) x 0.506GHz x 2SC x 2IOB M8000: 61GB/s = (4B(in)+4B(out)) x 0.480GHz x 2SC x 2Oberon x 4IOU M9000 (32 CPUs): 122GB/s = (4B(in)+4B(out)) x 0.480GHz x 2SC x 2Oberon x 8IOU M9000 (64 CPUs): 244GB/s = (4B(in)+4B(out)) x 0.480GHz x 2SC x 2Oberon x 16IOU
  81. 81. 81 Bandwidth and Streams Benchmark •The theoretical system bandwidth at peak time is calculated by multiplying the bus width by the bus frequency between the system controller and memory access controller. •The theoretical I/O bandwidth at peak time is calculated by multiplying the bus width by the bus frequency between the system controller and PCI bridge. In GB/s Theoretical System Bandwidth Snoop Bandwidth Streams (Triad) Streams (Copy) Theoretical I/O Bandwidth M4000 32 129 12.7 12.6 8 M5000 64 129 25.3 24.8 16 M8000 184 245 69.6 60.3 61 M9000-32 368 245 134.3 114.9 122 M9000-64 737 245 227.1 224.4 244
  82. 82. 82 High-end System Cabinets M8000 System M9000 System M9000 System Front Rear CMU Fan Expansion Cabinet XB PSU DVD IOU XSCF AC input selector Clock Board IOU Fan Base Cabinet ** Rack for I/O Expansion **Rack for I/O Expansion ** Rack for I/O Expansion is optional and applies to M8000, M9000 720 1800 1200 1200 1800 810
  83. 83. 83 COMMONALITIES (M8000 vs. M9000) • CPU module • CPU memory unit (CMU) • I/O unit (IOU) • FAN trays ( Fan_A (3)) • Power Supply Unit (PSU) • XSCF board • Operator Panel board • 2.5” SAS disk • DVD, DAT • PCI cassette • Memory DIFFERENCIES (M8000 vs. M9000) • Max. number of CMU and IOU > M8000: 4CMU/IOU -> Max. 16 domain > M9000: 8CMU/IOU in one cabinet -> Max 24domain • XB/CLK > M8000: XB/CLK exists on BP_A, Two clock source, but one clock ASIC > M9000: XB is in XBU, clock is in CLKU (Two clock source, and two clock ASICs) • DDC for XB/CLK > M8000: Exist as a separate unit, DDC_A > M9000: Integrated to XBU/CLKU • Expansion cabinet for M9000 only • Internal rack space > M8000: 12U available > M9000: None • Power subsystem/Cooling subsystem > Different number of PSU/power cables > Different kind/number of Fans M8000 and M9000 Comparison
  84. 84. 84 M8000/M9000 System's RAS Management •Redundant interconnect •Redundant Service Processor •No Single Failure can Prohibit Reboot •Most FRU's are Hot-Swappable
  85. 85. 85 M8000 System 1-4 IO Units (IOU) (4 Disk Bays Each) 1-4 CMU Boards 19” Rack Space Fan Trays PSU Fan Trays XSCFU_B#0 XSCFU_B#1 Rear View Front View Rack space for Dual Power Grid Option
  86. 86. 86 M9000-32 System IO Boards (4 Disk Bays Each) Fans PSU’s Crossbar Boards Clock Boards XSCF Boards DAT DVD Front View
  87. 87. 87 M9000-32 System IO Boards (4 Disk Bays Each) Fans CMU Boards Back View
  88. 88. 88 M9000-64 System IO Boards (4 Disk Bays Each) Fans PSU’s Crossbar Boards Clock Boards XSCF Boards DAT DVD Front View Clock Boards XSCF Boards Crossbar Boards DAT DVD IO Boards
  89. 89. 89 M9000-64 System (no cables) Crossbar Boards Clock Boards XSCF Boards Crossbar Boards Clock Boards XSCF Boards
  90. 90. 90 M9000-64 System IO Boards FansFans CMU Boards Back View CMU Boards IO Boards
  91. 91. 91 M9000 + External Power Cabinets External Power Cabinet External Power Cabinet
  92. 92. 92 M9000 + External Power Cabinets External Power Cabinet External Power Cabinet
  93. 93. 93 Backplane BP_A (for M8000) Active Backplane
  94. 94. 94 Backplane BP_B (for M9000) Passive Backplane
  95. 95. 95 M8000/M9000 High-End Building Blocks • Max 4 CPU sockets / 8 cores • Can mix (2) SPARC64 VI and (2) SPARC64 VII/VII+ on same CMU • 2x 16 DIMM slots (1, 2, 4 and 8 GB) • 16, 32, 64, 128, and 256 GB memory expansions • Memory mirror option only in Uni-XSB mode 2x 16 DIMM 4 CPU Sockets Red. DC-DC Converter CPU/Memory Unit (CMU)
  96. 96. 96 Side note: Getting Our Terms Straight Uni-XSB: When the system board is not logically divided, it consists of a single XSB Quad-XSB: When the system board is logically divided, it consists of four XSBs
  97. 97. 97 M8000/M9000 High-End Building Blocks • 1x IOU per CMU or 1x IOU per multiple CMU • 2x 4 PCIe slots • Optional base IO card (2x SAS, 2x GbE) using PCIe slot • PCIe slot assignment to 4x XSB • Link up to 4 full I/O expansion boats (link cards use PCIe slots) I/O Unit (IOU) 2x2 Disk Bays 2x4 PCIe Slots
  98. 98. 98 PCICS: PCI Cassette for M8000/M9000 IOU Carrier main body (metal) Moveable carrier plate (plastic) Operating handle Card retainer (5 total, 3 types) Circular area for rotating the retainer In PCICS, PCIexpress Short dedicated cassette provided
  99. 99. 99 • • • CPU0 CPU1 CPU 2 CPU3 MAC0 MAC1 MAC2 MAC3 4 DIMMs 4 DIMMs 4 DIMMs 4 DIMMs M8000/M9000 System Board Block Diagram All Buses are 4B Uni-Directional Paths with Multiplexed Address and Data Plus Control and ECC in Each Direction SC0 SC1 SC2 SC3 4B per path @1.07 GHz 2 paths each direction ~5 to 6 GB/sec/way peak (~30-35% data 65-70% broadcast address and control) to/from XB way-0 to/from XB way-1 to/from XB way-2 4B per path @ 0.53GHz 1 path each direction to/from XB way-3 4B per path @1.07GHz 1 path each direction 4 DIMMs 4 DIMMs 4 DIMMs 4 DIMMs
  100. 100. 100 35 GB/sec peak M8000/M9000 System Board Block Diagram 8.7 GB/S ~20-24 GB/sec peak ~40 GB/sec • ~10 GB/sec peak • All Buses are 4B Uni-Directional Paths with Multiplexed Address and Data Plus Control and ECC in Each Direction CPU0 CPU1 CPU 2 CPU3 MAC0 MAC1 MAC2 MAC3 4 DIMMs 4 DIMMs 4 DIMMs 4 DIMMs SC0 SC1 SC2 SC3 4 DIMMs 4 DIMMs 4 DIMMs 4 DIMMs 5~6 GB/sec peak
  101. 101. 101 PCIexpress Short Slot8 PCIexpress Short Slot8 PCIexpress Short Slot8 PCIexpress Short Slot8 PCIexpress Short Slot8 PCIexpress Short Slot8 PCIexpress Short Slot8 PCIexpress Short Slot8 M8000/M9000 I/O Board Block Diagram Base IO Board PCIe bridge PCIe bridge PCIe Switch Internal DVD Drive Internal DAT Drive Internal Drives EXP DVD DA TGBit Ethernet SAS PCIe to PCI- X bridge IOU to/from one way – 2 XB’s EXP in cabinet 5-6 GB/sec peak • PCIe Switch PCIe Switch PCIe Switch to/from one way – 2 XB’s
  102. 102. 102 IOUA (Base IO Card) > To IOU Internal HDD when installed in PCI#0 or #4 > To DVD/DAT drive Via SAS exp when installed in PCI 0,2,4 or 6 Base IO Board Internal DVD Drive Internal DAT Drive Internal Drives GBit Ethernet SAS PCIe to PCI- X bridge
  103. 103. 103 ~43 GBps M8000/M9000 Boards 0.53 Ghz ~40 GBps System Board I/O Board Paths are 4 bytes each way and carry both address & data ~1/3 of traffic is data 1.07 GHz 24 Cables to other Xbars 0.5–1 GHz ~48 GBps ~22GBps XB Board X8 X8 X8 X8 X8 X8 X8 X8 PCIexpress slot PCIexpress slot PCIexpress slot PCIexpress slot PCIexpress slot PCIexpress slot PCIexpress slot PCIexpress slot Processor Memory Controller Processor Memory Controller Memory Controller Processor Processor Memory Controller ~22 GBps System Controller System Controller System Controller System Controlle r ~5.5 GBps PCIe x8 bridge PCIe x8 bridge 8x8 Crossbar 8x8 Crossbar 8x8 Crossbar 8x8 Crossbar ~49 GBps 8 DDR2 DIMMs 8 DDR2 DIMMs 8 DDR2 DIMMs 8 DDR2 DIMMs PCIe Switch PCIe Switch PCIe Switch PCIe Switch
  104. 104. 104 Detailed M9000 Block Diagram
  105. 105. 105 Quad-XSB Configuration System Board Processor Memory Controller 8 DDR2 DIMMs Processor Memory Controller 8 DDR2 DIMMs Processor Memory Controller 8 DDR2 DIMMs Processor Memory Controller 8 DDR2 DIMMs Quad-XSB 00-3 Quad-XSB 00-2 Quad-XSB 00-1 Quad-XSB 00-0 I/O Board PCIe slot 0 PCIe slot 1 PCIe slot 2 PCIe slot 3 PCIe slot 4 PCIe slot 5 PCIe slot 6 PCIe slot 7 PCIe x8 bridge PCIe x8 bridge PCIe Switch PCIe Switch PCIe Switch PCIe Switch
  106. 106. 106 M8000/M9000 Backplane Interconnect XB 2 XB/way 4 ways I O U I O U I O U IOU M9000 C M C M CMU XB 2 XB/way 4 ways I O U I O U I O U IOU C M C M CMU 368 GB/sec peak M8000 C M CMU I O U I O U I O U IOU XB 2 XB/way 4 ways 184 GB/sec peak
  107. 107. 107 Scaleable Interconnect M8000 C M C M CMU I O U I O U I O U IOU XB 2 XB/way 4 ways XB 2 XB/way 4 ways I O U I O U I O U IOU M9000 C M C U CMU I O U I O U I O U IOU XB 2 XB/way 4 ways C M C M CMU XB 2 XB/way 4 ways I O U I O U I O U IOU M9000 C M C M CMU 368 GB/sec peak 737 GB/sec peak 184 GB/sec peak
  108. 108. 108 Operator Panel LED, Button or Switch Name Icon Color State/Setting Description Power Green On Off System Power is on System power is off Standby XSCF Green On Blinking Indicates the system can be powered on XSCF on-Indicates that the system can be powered on. System initialization/power on processes in progress Off XSCF off-Indicates that the system cannot be powered on. (circuit breakers are off) Check Amber On Blinking Indicates the operator panel is the maintenance target device. Indicates these conditions; ● Error detected. System startup disabled ● Power failure occurred Off Indicates the operating status of the system. Indicates one of three conditions; ● Normal operating state ● Circuit breakers are switched off ● Power failure occurred POWER Button Less than 4 secs. ● Regardless of the MODE switch state, powers on all domains ● Skips processing for waiting for facility and power-on warm up. 4 secs. Or longer (Service) Performs one of these functions; ● If power-on is being performed, the power-on processing is cancelled and the system is powered off. ● If the system is being powered off, the operator of the power switch is ignored. The power-off process continues until the system is powered off.
  109. 109. 109 Operator Panel LED, Button or Switch Name Icon Color State/S etting Description MODE Switch (1) Locked ● Normal operator made. (recommended setting for day-to-day operators) ● System can be powered on. ● System cannot be powered off. ● Key can be removed. MODE Switch (2) Service ● Service should be provided. (switch must be set to service mode) ● Key cannot be removed. ● System can be powered on and off with the POWER button. ● Prevents unintentional power-on.
  110. 110. 110 M9000-64 Cabling XB Data : 48 cables XB CLK : 8 cables CLK Data : 2 cables CLK CLK : 4 cables XSCF Data : 2 cables Total : 64 cables
  111. 111. 111 M9000-64 Cabling
  112. 112. 112 XBU #8 to #15 M9000-64 Cabling: XB M9000 Base Cabinet M9000 Ex. Cabinet Yellow Pink Brown Green Orange Blue Colored Tie Wraps XBU #0 to #7 1.3m Data Cable 1.0m CLK Cable SPOF
  113. 113. 113 M9000-64 Cabling: XB M9000 Base Cabinet M9000 Ex. Cabinet
  114. 114. 114 M9000-64 Cabling: CLKU CLKU#0 CLKU#1 out out in out out in M9000 Base Cabinet M9000 Ex. Cabinet CLKU#2 CLKU#3 out out in out out in Data Cable CLK Cable SPOF
  115. 115. 115 M9000-64 Cabling: XSCF XSCFU_B#0 XSCFU_B#1 M9000 Base Cabinet M9000 Ex. Cabinet XSCFU_C#0 XSCFU_C#1 Data Cable
  116. 116. 116 Airflow Front Back M8000 M9000
  117. 117. 117 Dual Power Feed, 3-Phase Power for M8000 Single phase input & Dual Power Feed 3-phase input (Delta) & Dual Power Feed 3-phase input (Star) & Dual Power Feed Single Phase Input (standard) 3 Cables Base Cabinet 12U DPF Option 3 Phase Input (DPF) Base Cabinet 12 U Power Cabinet 3 Phase (Delta ) DPF 3 Phase Input (DPF) Base Cabinet 12 U Power Cabinet 3 Phase (Star) DPF 7 U Single Phase Input (DPF Option) * Customer must have their electrician make 3-phase cables Descriptions Remarks 1 Single phase DPF unit 7U rack unit for M8000 base cabinet, field installable 2 M8000 base cabinet + power cabinet for 3- Phase (delta) DPF M8000 Base cabinet + Power cabinet for 3-Phas (delta)/ Dual Power Feed Power, 2 lines, Delta (208V) is mainly used in Japan and North America. 3 M8000 base cabinet + power cabinet for 3- Phase (star) DPF M8000 Base cabinet + Power cabinet for 3-Phase(star)/ Dual Power Phase, 2 lines, Star(415V) is mainly used in EU region.
  118. 118. 118 Dual Power Feed, 3-Phase Power for M9000, Single Cabinet 3-phase input & Dual Power Feed 3 Phase Input (DPF) Base CabinetPower Cabinet 3 Phase (Delta or Star) DPF Single phase input & Dual Power Feed Base CabinetPower Cabinet DPF Option Single Phase Input (DPF Option) Single Phase Input (Standard) 5 Cables * Customer must have their electrician make 3-phase cables Descriptions Remarks 1 Power cabinet for single phase DPF Power cabinet, field installable 2 M9000 base cabinet + power cabinet for 3- Phase (delta) DPF M9000 base cabinet for 3-phase(delta)/ Dual Power Feed, 2 lines, Delta(208V) is mainly used in Japan and North America. 3 M9000 base cabinet + power cabinet for 3- Phase (star) DPF M9000 base cabinet for 3-phase(star)/ dual Power Feed, 2 lines, Star(415V) is mainly used in EU region.
  119. 119. 119 Dual Power Feed, 3-Phase Power for M9000, Dual Cabinets 3 Phase Input (DPF) 3 Phase Input (DPF) 3-phase input, Dual Power Feed Extension Cabinet Base CabinetPower Cabinet 3 Phase (Delta or Star) DPF Power Cabinet 3 Phase (Delta or Star) DPF Single Phase Input (DPF Option) Single Phase Input (Standard) 5 Cables Single phase input, Dual Power Feed Base CabinetPower Cabinet DPF Option Extension Cabinet Power Cabinet DPF Option Single Phase Input (Standard) Single Phase Input (DPF Option) * Customer must have their electrician make 3-phase cables Descriptions Remarks 1 Dual power cabinet for single phase DPF 2x Power cabinet, field installable 2 M9000 base cabinet + expansion cabinet for 3- Phase (delta) DPF M9000 Base cabinet + Power cabinet for 3-Phase(delta)/ Dual Power Feed Power, 4 lines, Delta(208V) is mainly used in Japan and North America. 3 M9000 base cabinet + expansion cabinet for 3- Phase (star) DPF M9000 Base cabinet + Power cabinet for 3-Phase(star)/ Dual Power Phase, 4 lines, Star(415V) is mainly used in EU region.
  120. 120. 120 Service Area: M8000 800 (31.5) 800 (31.5) 1260 (49.6) 750 (29.5) M8000 Front M8000 Front PowerCabinet 1054 (41.5) Sun Rack 900 (35.4) 900 (35.4) 850 (33.5) 1260 (49.6) Front Units in mm () in inches
  121. 121. 121 Service Area: M9000 M9000 900 (35.4) 900 (35.4) 850 (33.5) 1260 (49.6) Front M9000 Front PowerCabinet 1154 (45.4) Units in mm () in inches M9000 Expansion Cabinet Front PowerCabinet PowerCabinet 2282 (90.0) M9000 Front Expansion Cabinet 1674 (65.9) Only the M9000-64 requires a raised floor with ventilation holes
  122. 122. 122 <Insert Picture Here> I/O Expansion Unit
  123. 123. 123 External IO Expansion Unit • Common unit between mid-range and high-end servers • Rack mountable (4U) • All cabling rear accessible • Redundant and hot-swappable PSU’s and fans • Serial link connection between host using link cards • Copper cable (3m) and optical fibre cable (up to 25m) • Two IO boats with hot-pluggable PCI slots • 2 Boats with PCI-X slots • 2 Boats with PCIe slots • One boat with PCIe slots and one boat with PCI-X slots
  124. 124. 124 I/O Box: FRUs and PCI Data Path 32 Lane PCIe Switch PLX PEX8532 Switch Bridge Bridge Bridge NEC PCI-X to PCIe Bridge UPD720400 8 Lane PCIe Uplink 6 PCI-X Slots Link 0 Slot 1 Slot 2 Slot 4 Slot 5 Slot 6 Slot 3 Switch Switch Switch 8 Lane PCIe Link 0 Slot 1 Slot 2 Slot 4 Slot 5 Slot 6 Slot 3 6 PCIe Slots
  125. 125. 125 What do you mean “bottleneck”? •8 lane(8x) PCIe is approximately 4 times faster than what we support with 133Mhz 64 bit PCI-X cards (32 Gb -vs- 8 Gb) •Advantage to the PCIe architecture is that it is point to point, so issues around bus length are greatly reduced •Current cards are 2x and 4x Adapter Expected (Gb/s) Theoretical (Gb/s) Quad GbE 2.8 4 10 GbE JumboFrame 9 20 10 Gb iSCSI w/TOE 9 20 4/2/1 Gb FC Dual Port 1 1.6 8/4/2 Gb FC Dual Port 1.8 3.2 IB 4x Dual Port 16 20 IB 8x (DDR) 32 40 40 GbE 35 80
  126. 126. 126 I/O Boat
  127. 127. 127 PCI Cassette for Boat in I/O Box In PCICS, PCIexpress Short dedicated cassette provided Carrier main body (metal) Moveable carrier plate (plastic) Carrier slot keyholes Operating handle Card retainer (5 total, 3 types) Circular area for rotating the retainer
  128. 128. 128 Inserting and Removing a Carrier from the I/O Boat a. Pulling the carrier handle raises PCI card out of PCI slot connector b. Pushing the carrier handle lowers PCI card into PCI slot connector c. Lever locks the handle in the extended position. Push the lever to the left to unlock the carrier handle.
  129. 129. 129 I/O Box: Link Card • Add-in card form factor • 8 lane PCIe path • Supports cable with sideband serial link • Does store and forward of I2C IO Box management traffic over sideband serial link • Bridge controller provides reliable transport protocol over sideband serial link • Supports an optional (non-standard) interrupt pin for host SP notifications
  130. 130. 130 Host to I/O Expansion Unit Link •Host to I/O Expansion Unit connectivity is obtained through either – (1) low height copper link card kit – (2) full height fibre link card kit •Host can have management control via side-band signals
  131. 131. 131 I/O Box: Interconnect Cable/Fibre • Copper cable - follows PCIexpress standard with sideband signals • Maximum length 5 meters • Optical fibre - follows PCIexpress standard with sideband signals • Maximum length 25 meters • Removal of cable will result in domain panic
  132. 132. 132 Copper Link Kit •Each half height PCIe based copper link card has 1 interconnect (i.e. cable plug) which carries data over a 5 meter distance • 8 lane PCIe (data) • Bidirectional • serial (mgmt)
  133. 133. 133 Fibre Link Kit •Each full height PCIe based fibre link card has: – 1 Transmit (TX) Port – 1 Receive (RX) Port •Transceivers are/use: – 850nm wavelength – 50/125 MM Fibre – IEC 60825 & CDRH Class 1M laser eye safe
  134. 134. 134 Front View - Power Supplies (A195) •Two 600W 120-240V Hot Swap PS's – Redundant power & cooling (can run w/ 1 PS) •Power supply and chassis status LED's PSU 1 600 Watts 120/240 VAC Fault Power Overtemp Locate PSU 0 600 Watts 120/240 VAC AC Power Ready-to-remove Fault /Locate OK
  135. 135. 135 Boat Fault/Locate Rear View – I/O Expansion Boats •Two independent I/O “boats” – Mix/match either PCIe or PCI-X boat – Dual hosting is not supported (discussed later) Chassis Locator Chassis Fault Chassis Power Chassis Overtemp PCIe link To Host Ready to Remove Boat Ok
  136. 136. 136 PCI cards for M8000 - M9000 •PCI Cards – Base I/O card (IOUA) can be populated into #0, #2, #4, #6, depending on hardware implementation. – Link card can be populated into #1, #3, #5, #7, to avoid conflicting Base I/O card. IOU PCIe Slot # Base I/O Card Allowed? Link Card Allowed? Available Resources In IOU Via Link Card #0 Yes Disk#0, #1,DVD/DAT,2xGbE #1 Yes I/O Boat #2 Yes DVD(*)/DAT(*), 2xGbE #3 Yes I/O Boat #4 Yes Disk#0, #1, DVD/DAT, 2xGbE #5 Yes I/O Boat #6 Yes DVD(*)/DAT(*),2xGbE #7 Yes I/O Boat
  137. 137. 137 IOUA and available resources in IOU • When populated into #0, #4: Disks, and DVD(*)/DAT(*) are available • When populated into #2, #6: DVD(*)/DAT(*) are available (*) One DVD and one DAT (option) are installed per base cabinet and expansion cabinet(M9000). And, DVD or DAT drive are exclusively connected to one Base I/O card.
  138. 138. 138 IO Box Maximum Connections Max I/O Box on M9000-64 restricted to same quantity as M9000-32 due to limited testing. I/O Box Configuration Rules Mixing PCIe IO-Boat and PCI-X IO-Boat in an IO-Box is supported. A Link option must be ordered per IO-Boat. M4000 M5000 M8000 M9000-32 M9000-64 Max IOU 1 2 4 8 16 Max I/O Boxes 2 4 8 16 Max I/O Boats 4 8 16 32
  139. 139. 139 Maximum IO Expansion Units for M4000/M5000 • Concept – Up to 2x IO Boxes per IOU for M4000 & M5000 IO-Box IOU PCIe Link CardPCIe PCIe PCIe PCIe PCI-X Link cards can be populated into any PCIe slot IO-Box The number of PCI slots per 18 slots IOU : 25 slots IOU: 1 (PCI-X) IOU-Box: 2x 12 (see note-1) Notes-1: As for 12 slots per IO-Box, either PCIe or PCI-X can be selected in 6 slots. Link Card Link Card Link Card PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe Max I/O Boxes Max PCIe Slots Remarks M4000 2 25 1 IOU x 25 slots M5000 4 50 1 IOU x 25 slots
  140. 140. 140 IO Box Configuration for M4000/M5000 M4000: Maximum of 2x IO Boxes (M4000: 6U, IO Box: 4U) M4000 IO-Box IO-Box IOU #0 x2Link 19” Rack M5000 IO-Box IO-Box IO-Box IO-Box IOU #1 IOU #0 x2Link x2 19” Rack M5000: Maximum of 4x IO Boxes (M5000: 10U, IO Box: 4U) Notes: The physical mounting location of IO-Box is not fixed.
  141. 141. 141 Number of IO Expansion Units for M8000/M9000 • Concept – Up to 2x IO Boxes per IOU IO-Box IO U PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe PCIe Link Card PCIe PCIe IO-Box PCIe PCIe PCIe PCIe PCIe PCIe #0 #1 #2 #3 #4 #5 #6 #7 Notes-1: As for 12 slots per IO-Box, either PCIe or PCI-X can be selected in 6 slots. Link Card Link Card Link Card Max I/O Boxes Max PCIe Slots Remarks M8000 8 112 4 IOU x 28 slots M9000-32 16 224 8 IOU x 28 slots M9000-64 288 Limited to testing
  142. 142. 142 IO Box Configuration for M8000 & M9000 Notes: The physical mounting location of IO Box is not fixed. 19” Rack IO- Box x 2 x2 x2 x2Link Base Cabinet Rear View M8000: Maximum of 8x IO Boxes (IO Box: 4U) IOU #3 # 2 #1 #0 19” Rack Base Cabinet19” Rack Rear View M9000: Maximum of 16x IO Boxes (IO Box: 4U) Link IO U #6 #4 IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box IO- Box x 2 x2 x2 x2 IOU IOU IOU IO U IO U IO U #2 #0 IO U #7 #5 IO U IO U IO U #3 #1
  143. 143. 143 IO Box Configuration for M9000 IO-Box IO-Box IO-Box IO-Box IO-Box IO-Box IO-Box Expansion Cabinet 19” Rack IO-Box IO-Box IO-Box IO-Box IO-Box IO-Box IO-Box IO-Box Rear View x1 x1 x1 x1Link Base Cabinet19” Rack IO-Box IOU #14 M9000: Maximum of 16x IO Boxes Configuration (IO-Box: 4U) Notes: The physical mounting location of IO Box is not fixed. IOU #12 IOU #10 IOU #8 IOU #15 IOU #13 IOU #11 IOU #9 IOU #6 IOU #4 IOU #2 IOU #0 IOU #7 IOU #5 IOU #3 IOU #1 x1 x1 x1 x1Link
  144. 144. 144 I/O Box SCF Software • IO Manager Software on SCF • Largely platform-independent, could be used on OPL, Supernova or otherplatforms. • Includes all the functionality required for IO Box management. • Discovers IO Boxes and FRUs (part numbers, serial numbers, etc) when PCIe slots are powered on. • Can collect environmental, voltage, status information from IO Box FRUs. • Receives interrupts when the IO Box detects errors
  145. 145. 145 I/O Box Software SCF Software Hardware IO Box BUI IO Box CLI Commands Error Reporting SunMC Agent Module Wrapper libsmbus IO Manager (libcharon)
  146. 146. 146 I/O Box CLI •The only IO Box CLI command on SCF is ioxadm    Usage: ioxadm [­h]       or: ioxadm [­fpv] list [TARGET]       or: ioxadm [­fpv] env [­elt] [TARGET]...       or: ioxadm [­fpv] reset TARGET       or: ioxadm [­fpv] resetpci TARGET       or: ioxadm [­fpv] poweroff TARGET       or: ioxadm [­fpv] poweron TARGET       or: ioxadm [­fpv] setled [­n|­f|­S|­F] TARGET LED       or: ioxadm [­fpv] locator [­n|­f] TARGET •TARGET is either a host pci slot location or a box/fru – PCI Slot: IOU#0-PCI#3 – FRU: iox@<serial id>[/fru] • iox@3B5A – The IO Box with serial id ending in 3B5A • iox@3B5A/ps1 – PSU #1 in IO Box • iox@3B5A/iob0/link – Link card in IO Boat #0 in IO Box
  147. 147. 147 For More Information search.oracle.com or oracle.com/us/products/servers-storage/servers/sparc-enterprise Servers
  148. 148. 148

×