Univ. of Wisconsin: Solving the HPC Data Deluge


Published on

In this presentation from the Dell booth at SC13, Scott Nolin from the University of Wisconsin describes how Big Data gets handled for HPC at the University.

"As science drives a rapidly growing need for storage, existing environments face increasing pressure to expand capabilities while controlling costs. Many researchers, scientists and engineers find that they are outgrowing their current system, but fear their organizations may be too small to cover the cost and support needed for more storage. Join these experts for a lively discussion on how you can take control and solve the HPC data deluge."

Watch the video presentation: http://insidehpc.com/2013/12/03/panel-discussion-solving-hpc-data-deluge/

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Univ. of Wisconsin: Solving the HPC Data Deluge

  1. 1. SSEC Storage Systems Panel: Solving the Data Deluge Scott Nolin 20 November 2013 1
  2. 2. About SSEC The University of Wisconsin Space Science and Engineering center (SSEC) is a research and development center focusing on geophysical research and technology to enhance our understanding of the atmosphere of Earth, the other planets in our Solar System, and the cosmos. Major SSEC Initiatives §  atmospheric studies of Earth and other planets §  interactive computing, data access, and image processing §  spaceflight hardware development and fabrication Noted Scientific Work §  satellite-based and other weather observing instruments §  remote sensing applications in earth and atmospheric science §  spaceflight instrumentation §  planetary meteorology §  data analysis and visualization §  diagnostic and numerical studies of the atmosphere 2
  3. 3. What Shapes our Storage Solutions? §  The SSEC contains many groups and projects, often with different funding sources. Individual groups and projects ultimately control their own expenditures. §  §  The Technical Computing Group provides an infrastructure to share and combine these resources. Atmospheric research via satellite meteorology is the major focus of SSEC. §  §  This has large data needs. Files are typically fairly large – we use 100MB files as a generic representation of a lot of our data. Most projects are research, and not strictly operational so high availability is usually not required, but data integrity is critical. 3
  4. 4. Uniform Components,Varied Solutions NAS MDS/MDT FDR-­‐10 OSS  1 FDR-­‐10 OST0  OST1  OST2 Split  M ode SAS   Cables SAS   Cables OST3  OST4  OST5 OSS  2 FDR-­‐10 4
  5. 5. High Performance Storage Infrastructure SSEC has recently created a generally available storage infrastructure, largely focused on Lustre. Making large file systems available generally, and not just on compute clusters has changed how people work with large data. Key components include: §  Infiniband §  §  Lustre Routers §  §  Allow connecting the infiniband resources (storage, compute clusters) to 10G and 1G ethernet systems. Patchless Lustre Clients §  §  Once you have taken the step to infiniband infrastructure (which was required by some of our NWP needs) it’s hard to pay more for lower performance with ethernet This is particularly useful for extending lustre filesystems to non-cluster nodes. Linux is the main focus here, our ‘clients’ are what many would consider servers, typically ethernet connected. BETA - Providing a mountable file system aggregating all resources (NAS, Gluster, Lustre) as a single file system to desktops. §  Work in progress – custom application in house by a research group 5