Lustre User LINES OF TEXTSUBTITLE WITH TWO                  Group 2009IF NECESSARYLustre as the Core of a Data CentricBest...
A recent conversation “Its not computation Im worried about, thats been solved, its storing data, accessing it, and moving...
Processing power doubles every 2years, but not for HPC...              Peak FLOPS Per Rack        In 2005:                ...
Data ExplosionSource: IDC                  Lustre Users Group 2009   4
Factors driving the HPC data tsunami• Increased sensor resolution    -Cameras, Confocal Microscopes, CT Scanners, Sequence...
Like computers themselves, the devicescreating these datasets have proliferated• From 1 per institution, to 1 per departme...
Oh, all the previously accrued data needs to bestored too...     Protein Database Example                                L...
Conclusion• You can only compute the data as fast as you can move it• HPC workflows will become performance bound by the  ...
So data access time will dominate HPCworkflow        Time to Load Data   Time to Compute           Time to Store Data 2005...
Sun end-to-end infrastructure optimized to     accelerate data centric HPC workf ows                                     l...
High Bandwidth, High Memory Petaflop-Scale HPC Blade System• Sun Blade X6275                                    New   New,...
Breakaway application performanceCompute-Intensive                            Sun Blade X6275 48 Node ClusterMCAE Applicat...
Sun Data Center 3456 IB Switch• Worlds Largest Infiniband Core Switch• Replaces 300 discrete InfiniBand  switches and thou...
Efficient Petascale ArchitectureConventional                                           Constellation                  s   ...
Sun Lustre Storage System              •    Scales to over 100 GBs/sec and capacity to petabytes              •    Acceler...
Sun Storage 7410 Unified Storage SystemBreakthrough NAS performance at low cost and powerwith Hybrid Storage Pool         ...
Scratch storage for a small HPC cluster                                                                            7410 sc...
Unprecedented Dtrace Storage Analytics• Automatic real-time visualization of  application and storage related workloads• S...
Video        Lustre Users Group 2009   19
Sun StorageTek Tape ArchiveThe greenest storage on the planet• Provides a massive on-line/near-line repository –  up to 70...
Putting it all together, theSun Constellation System• An open systems super computer designed to scale from  Departmental ...
More than any other computer  company, Sun gets itThis weeks “Nehalem” Press Releases:                                    ...
Summary• Data is taking over HPC• Sun has a wide array of complementary IP to Lustre that  can cope with that• Sun is lead...
THANK YOU
Upcoming SlideShare
Loading in …5
×

Lug best practice_hpc_workflow

234 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
234
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Lug best practice_hpc_workflow

  1. 1. Lustre User LINES OF TEXTSUBTITLE WITH TWO Group 2009IF NECESSARYLustre as the Core of a Data CentricBest Practice HPC WorkflowBob Murphy, Open StorageSun Microsystems
  2. 2. A recent conversation “Its not computation Im worried about, thats been solved, its storing data, accessing it, and moving it around.” - Rico Magsipoc, Chief Technology Officer for the Laboratory of Neuro Imaging, UCLA Lustre Users Group 2009 2
  3. 3. Processing power doubles every 2years, but not for HPC... Peak FLOPS Per Rack In 2005: 84 cores 500 GFLOPS In 2009: 768 cores 9 TFLOPS In 2011: 1,536 cores 172 TFLOPS Lustre Users Group 2009 3
  4. 4. Data ExplosionSource: IDC Lustre Users Group 2009 4
  5. 5. Factors driving the HPC data tsunami• Increased sensor resolution -Cameras, Confocal Microscopes, CT Scanners, Sequencers, etc Protein Database Example Lustre Users Group 2009 5
  6. 6. Like computers themselves, the devicescreating these datasets have proliferated• From 1 per institution, to 1 per department, to 1 per lab, to 1 per investigator Protein Database Example• Terabyte-sized datasets from a single experiment are now routine• 1TB/day/investigator• Plus more - and higher resolution - simulations per unit time Lustre Users Group 2009 6
  7. 7. Oh, all the previously accrued data needs to bestored too... Protein Database Example Lustre Users Group 2009 7
  8. 8. Conclusion• You can only compute the data as fast as you can move it• HPC workflows will become performance bound by the speed of the storage system Lustre Users Group 2009 8
  9. 9. So data access time will dominate HPCworkflow Time to Load Data Time to Compute Time to Store Data 2005 2009 2011 t Lustre Users Group 2009 9
  10. 10. Sun end-to-end infrastructure optimized to accelerate data centric HPC workf ows l Network Storage Archive StorageSun Storage 7000 Unified Storage System Sun StorageTek Tape ArchivesHigh Availability, Manageability, Shared Access Economic, Green, Long Term Data Retention Parallel Storage● Home Directories, Application Code ● Protection of IP Assets Sun Lustre Storage System● Input Data, Results Files ● SAM Storage Archive Manager HSM High Performance Parallel File System ● Ongoing Computation Lustre Users Group 2009 10
  11. 11. High Bandwidth, High Memory Petaflop-Scale HPC Blade System• Sun Blade X6275 New New, high density, dual node blade server 2X2 Nehalem-EP 4-core CPUs 2X12 DDR3 DIMM Slots, up to 192GB (8GB DIMMs) 2x1 Sun Flash Module (24GB SATA) 2x1 Dual Port QDR IB HCA• Sun Blade 6048 System 768 Nehalem-EP cores, 9TF, up to 9.2TB memory 96 X Gigabit Ethernet via NEM 96 X PCIe 2.0 Express Module x 8 interfaces Sun Blade X6275• Highest Compute Density in the Industry Sun Blade 6048 System Lustre Users Group 2009 11
  12. 12. Breakaway application performanceCompute-Intensive Sun Blade X6275 48 Node ClusterMCAE Applications Run Radioss Single Car Crash CodeFaster, with Less Power,Less Heat, Less Spacewith Integrated QDRInfiniband Performance Space Perf/Watt HP BL460c 2 x 3GHz Xeon E5456 Processors, SUSE Linux 3.2x 50% 3.2xSee Legal Substantiation Slides Lustre Users Group 2009 12
  13. 13. Sun Data Center 3456 IB Switch• Worlds Largest Infiniband Core Switch• Replaces 300 discrete InfiniBand switches and thousands of cables• Unparalleled scalability and dramatically simplifying cable management• Most economical InfiniBand cost/port• Prelude to “Project M9” doubling InfiniBand performance Lustre Users Group 2009 13
  14. 14. Efficient Petascale ArchitectureConventional Constellation s sCompute Rack Cluster RackClusters ng bli Le Ca af g in Sw Conventional bl Co witc Reduced Ca itc Cabling S re he Cabling he Infrastructure s sConventional IB Fabric Sun Datacenter Switch 3456• 300 Switches: 288 Leaf + 12 core • 1 Core Switch 300:1 Reduction• 6912 Cables: 3456 HCA + 3456 trunking • 1152 Cables 6:1 Reduction• 92 Racks • 74 Racks 20% Smaller footprint Lustre Users Group 2009 14
  15. 15. Sun Lustre Storage System • Scales to over 100 GBs/sec and capacity to petabytes • Accelerated deployment though pre-defined modules • Easy to size and architect configurations • Automated configuration and standard install process • Compelling price/performance with Sun components • Deployed by Sun Professional Services to ensure success Sun Lustre Storage System Performance ScalingPerformance 1 2 3 4 Number of HA OSS Modules 15 Lustre Users Group 2009
  16. 16. Sun Storage 7410 Unified Storage SystemBreakthrough NAS performance at low cost and powerwith Hybrid Storage Pool • Delivers over 1GB/s throughput • 2GB/s from DRAM • 280K IOPS • At ¼ the price of competitive systems • And 40% of the power consumption Sun Storage 7410 Unified Storage System• ZFS determines data access patterns and stores frequently accessed data in DRAM and FLASH• Bundles IO into sequential lazy writes for more efficient use of low cost SATA mechanical disks Lustre Users Group 2009 16
  17. 17. Scratch storage for a small HPC cluster 7410 scratch performance good forPre-configured Sun MCAE Compute single rack MCAE clusters Cluster Reduces install/operational complexity Expands market for HPC clusters -Many organizations “stuck on the desktop”Sun Storage 7410 Unified Storage System -Dont have expertise to roll their own clusterHigh Availability, Manageability, Shared Access● Home Directories, Application Code● Input Data, Results FilesOngoing Computation Scratch storageProtection of IP Assets “D-NAS” Lustre Users Group 2009 17
  18. 18. Unprecedented Dtrace Storage Analytics• Automatic real-time visualization of application and storage related workloads• Supports multiple simultaneous application and workload analysis in real-time• Analysis can be saved, exported and replayed for further analysis• Rapidly diagnose and resolve issues > How many Ops/Sec? > What services are active? > Which applications/users are causing performance issues? Lustre Users Group 2009 18
  19. 19. Video Lustre Users Group 2009 19
  20. 20. Sun StorageTek Tape ArchiveThe greenest storage on the planet• Provides a massive on-line/near-line repository – up to 70PB with little power or heat• SAM Storage Archive Manager maintains active data on faster disk and automatically migrates FC inactive data to secondary storage Data Mover Near Line Archive• All data appears locally and is equally accessible to multiple users and applications IB• Stores data in open formats (TAR) allowing technology refresh and avoiding vendor lock-in• Tape shelf life of about 30 years• Manage HPC data at the right cost, with the right performance, on the right media - and deliver it to applications at the right time Sun Lustre Storage System Lustre Users Group 2009 20
  21. 21. Putting it all together, theSun Constellation System• An open systems super computer designed to scale from Departmental Cluster to Petascale• Optimized compute, storage, networking and software technologies - and service - delivered as an integrated product• Integrated connectivity and management to reduce start-up, development and operational complexity• Technical innovation resulting in fewer components and high efficiency systems in a tightly integrated solution Easiest Path to Peta-Scale Lustre Users Group 2009 21
  22. 22. More than any other computer company, Sun gets itThis weeks “Nehalem” Press Releases: s-Press Re lease ms Product age Microsys tems, Inc . Ne two rk Syste g and Stor w Open etworkinSource: Sun Ne e, N S un An nounces en ce of Comput e Efficie ncy Converg em and Extr dy lash-Rea ts ce nt; New F Highligh Pe rforman ity, Cooli ng Mana geme eakaway eabil g, Manag with Br des A rchitectu re with N ew N etworkin ) Pro cessor 5 500 Serie s king, soft ware dvanc d Bla tel(R) Xe on(R etwor “Slow pipeseand slow storage have limited high performance computing mputing, n for d technologies Unveils A by the In en co systems ts an nce of op Su n d s Powere years. Bladesolution is to evolve the industrys view tofohigh performance new produc todayiency and s and The c nverge computing Server the mark e launched y ive effic said ws) to Fowler, executive mass I/O and networking,” A - NeJohn da to include 09, 9:10performanceWIRE)--Leading T am ED h 20 high Q: JAV ance, wit vice Webcast live or esday April 14, Tu president, SystemsU SS SINE Sun Microsystems. lif.--(B Group, stems, Inc. (NA S DA lication perform e launch L A RA , Ca sy ins in app e ga To view th s. SANTA C un Micro liver hug nd cloud ge s ystems, S g y that de ata centers a and stora ms strate ting for d /launch . n Netw ork Syste of compu w.sun.com in its Ope conomics http://ww im ize the e rma tion, go to y, to max duct info scalabilit d for more pro Lustre Users Group 2009 22
  23. 23. Summary• Data is taking over HPC• Sun has a wide array of complementary IP to Lustre that can cope with that• Sun is leading the industry in Data-Centric HPC by delivering differentiated customer value in balanced HPC systems, including compute, storage, and networking• Sun can uniquely deliver a complete end-to-end solution• You can also integrate parts of that solution into your existing workflow• Dont yell at your JBODs Lustre Users Group 2009 23
  24. 24. THANK YOU

×