In this presentation, Bryan Cote from Terascala describes the company's new Intelligent Storage Bridge technology.
"HPC workflows are most efficient when researchers have the data they need, when the need it. The Intelligent Storage Bridge acts like an air traffic controller, automatically moving large data sets on and of scratch using the maximum available network, compute, and storage resources. Once a researcher's work is complete, the data can then be transferred to more cost-appropriate enterprise storage for publishing and backup. Automating the workflow and placing the control in the hands of the researchers saves time and removes the need for additional training and data management expertise."
Learn more: http://www.terascala.com/product/intelligent-storage-bridge/
Watch the video presentation: http://wp.me/p3RLHQ-b3f
Introducing the Terascala Intelligent Storage Bridge
1.
2. Company Snapshot
- Founded in 2005 and based in Boston, MA
- Financial investment from Intel and several
venture firms
TARGET
MARKETS
- Manufacturing
- Life Sciences
CUSTOMERS
APPLICATION
BUSINESS
NEED
- Westinghouse
- Tradeworx
- product design
- genomics analysis
- long time to results
- many points of failure
- Oil and Gas
- Finance
- TGen
- Sandia
- Sold through Dell, NetApp
- Media
- Education
- Government
- Florida State University
- University of Minnesota
- high frequency trading
- seismic processing
- scratch data unprotected
- workflows not optimized
* PARTIAL LIST
- animation rendering
* PARTIAL LIST
- system too high touch
2
3. Our Product
•
TeraOS
»
•
The enabling software that is distributed across all
components to create a single, high-performance
storage appliance
Terascala Performance Engine (TPE)
»
»
•
The foundation of the storage appliance that
includes management and orchestration
software, performance and
analytics, monitoring, and tuning
Base performance is 3-7 GB/s
Terascala Performance Extension (TPX)
»
•
TeraView User Interface
»
•
Module that can be added to the Terascala
Performance Engine to linearly scale performance
by 3-4 GB/s
Provides a single unified view into the appliance to
see the performance and health of the system
Intelligent Storage Bridge
»
Fast and reliable data movement between
traditional enterprise and high performance storage
3
4. Sold Through Dell, EMC, NetApp
•
Terascala solutions are available through
• MORE
INFORMATION
DT-HSS
•
• MORE
INFORMATION
HPS Rack
Case Studies:
4
5. The Problem
• Large file sets are stranded in high-performance storage
• Researchers can’t access the data they need, when it’s needed
• Project data is unprotected and cannot be leveraged by the rest of the organization
X
HIGH-PERFORMANCE
STORAGE
X
ENTERPRISE
STORAGE
6. ISB Overview
The Intelligent Storage Bridge quickly moves large data sets between temporary and
permanent storage, ensuring data is preserved and available for other projects
Value
• First workflow manager
purpose-built for Lustre
environments
• Improves storage
efficiency
• Decreases time to
complete jobs
• Increases the number of
jobs that can be run
• Access data at any point
in the workflow
• Valuable project data is
protected and leveraged
Users
Specifications
• R&D and engineering
application users
• HPC / storage admins
• 1U rack
• Minimum base
configuration is 3 nodes
• Users define policies by
• 3 RU 2+1 redundancy
• Time
• Event
• File age
• Scales in 1U increments
• 48GBs memory
• 2 hot swappable 500GB
storage drives
• Single port FDR
InfiniBand
• 500 MB/s per 1U active
node
• Supports NFS
v3, CIFS, Lustre 1.88
• Compatible with NIS
7. How data is moved today
• Data movement is done
manually or automatically
Data transfer takes
5:41 hours with 1
server
• Manual data movement
6
» Usually requires expertise of
an admin
5
» Can take days to
copy, depending on number of
servers being used (see chart)
Hours
4
3
• Automatic data movement
2
» Requires scripts to automate
1
» Requires admin expertise and
constant monitoring
0
1
3
5
7
9
11
Nodes
Either method is prone to errors, requiring the process to be re-started from the beginning
8. Why the ISB is Unique
•
•
•
•
•
•
Appliance Solution
High Availability
Designed to offload Administration
Forms based, no scripting necessary
Monitoring integrated with TeraOS
Scales through the addition of additional
ISB Modules
10. Target Use Cases and Industries
• Use Cases
» Data ingest and re-ingest
» Job benefits from multiple runs
» Integrated with other clusters
• Industries
» Life Sciences – complex workflow
» Manufacturing – comparing multiple jobs
» Oil & Gas – iterative analysis of TB data sets
11. Use Case #1: Optimize Workflow
Genomics Sequence Workflow
1. Data from sequencers
(intermittent, MB/s)
2. Data availability
• Golden image to scale-out NAS
• Data to high performance
storage
3.
Data processing
• Data to apps ~3 GB/s or higher
4. Store results
5. Move scratch data to scale-out NAS
with Intelligent Storage Bridge
11
12. Use Case #2: Improve Scratch Efficiency
Automate Backup of TBs of Data and Improve
Scratch Efficiency
NFS
─
─
─
─
Intelligent
Storage Bridge
CIFS
─
─
NAS
Multiple users/applications sharing scratch at
multi-GB/s R/W
Working data backed up and protected on
NAS
Scalable performance and redundant
Move large data sets quickly via parallel
copy between PFS and different NAS
configurations
Role-based access controls (RBAC) ISB job
management
Fully managed by TeraView
Before
─ 200 users including 50+ power users running
ANSYS, Fluent, STAR
─ NFS solution can’t keep up
─ System crashes unpredictably
After
─ ANSYS 5x faster (61 to 11 hours)
─ No drop-outs
─ Tools to manage, analyze, predict and
optimize
12
13. Questions
• Additional Resources:
» Visit Terascala.com/resources for data
sheets, videos, white papers
» Blog: Terascala.com/blog
» Twitter: @Terascala_Inc
• How to buy:
» Contact Terascala
Bryan Cote
Product Management Director
Bryan.Cote@Terascala.com
M: 603-557-3568
Twitter: @BC_HPC
Twitter: @Terascala_Inc
Blog: Terascala.com/blog
Key Points:Terascala appliances are sold in every major market. Customers: this is a representative set of customers. We’re deployed at Fortune 500 companies, large and medium-size federal agencies, and a number of major universities.Applications: we’re focused on data-intensive applications that benefit from a high throughput storage appliance.Business need: we’ve heard from customers who have applications that may take as long as 30 hours or even several days to complete, which affects productivity and time to market. Companies may bring in proprietary solutions to handle scratch data that can run at a much higher performance. This meets the needs of the application in terms of throughput, but it comes at a price. That price is that the scratch data volume is large and unprotected—it’s not the typical NAS storage. These scratch storage systems are:high touch and have high administrative overhead that requires specialized expertisethey are often prone to having many points of failureworkflows are not optimizedscratch data then becomes an island that is not connected to the rest of the IT infrastructure. It satisfies the applications, but it doesn’t present an optimized workflow.
Key Points:TeraOS brings everything together so all of the components operate as an appliance with high availability. Without TeraOS, there will be many points of failure and a high cost of administration requiring in-house expertise to manage it. TeraOS removes these complexities. The performance comes from the Terascala Performance Engine. The core TPE has the hardware and software to deliver multiple GB/s to the application. The TPE provides all of the management, analytics, and tuning necessary for optimal performance across a variety of workloads. To get more performance, add more performance extension (TPX) modules. These can be scaled modularly with no limit—you can run 50 GB/s if need be. As modules are added, they operate under the umbrella of TeraOS.TeraView is a single pane of glass that let’s users look at the health and performance of system.All of the high-performance appliances on the marketplace sold by Dell, EMC, and NetApp use TeraOS in this unified interface—consistent across all appliances.
Key Points:Use the links provided to visit partner websites for more product information.Additional resources are available on terascala.com.