Sept 24 NISO Virtual Conference: Library Data in the Cloud

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Big Data Processing
in the Cloud
Zhiwu Xie and Collin Brittle

Data are abundant;
(re)usable data are scarce.
2
- Jane Silverthorne
Deputy Assistant Director, NSF

3
Outline
• Data, big data, and the library
• Why should the library process big data? A case
study: SEB
• How is thi...

Data, Big Data, and Library
• Research data as the first class citizen
• Data management mandates, OSTP memorandum, …
•...

The Big Data Long Tails
• Small number of large data sets as the head; large
number of smaller, messier data sets as the...

The Libraries’ Role
• Large data sets and grand challenges -> National
infrastructure, e.g., XSEDE
• Smaller, messier -...

SEB: A Case Study
7

Signature Engineering Building
8

Signature Engineering Building
9

Sensors
10

Smart Infrastructure
11

Data Sharing
• Encourage exploratory and multidisciplinary research
• Foster open and inclusive communities around
• mo...

Characterization
• Compute intensive
• Storage intensive
• Potentially bandwidth intensive
• On-demand
• Scalability ...

Compute Intensive
• About 6GB raw data per hour
• Must be continuously processed, ingested, and
further processed
• Us...

Storage Intensive
• SEB will accumulate about 60TB of raw data per year
• To facilitate researches on long-term effects,...

(Potentially) Bandwidth Intensive
• What if hundreds of researchers around the world
each tried to download hundreds of ...

On Demand
• Explorative and multidisciplinary researches cannot
predict the data usage a priori
17

Scalability
• How to deal with these challenges in a scalable
manner?
18

Big Data + Cloud
• Affordable
• Elastic
• Scalable
19

A Data Reuse Scenario
• Validate cross-disciplinary research hypothesis, e.g.,
find a novel vibration pattern against al...

Thank You!
• Questions? Comments?
• zhiwuxie@vt.edu
21

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Sept 24 NISO Virtual Conference: Li... by National Informat... 1092 views
Sept 24 NISO Virtual Conference: Li... by National Informat... 944 views
Sept 24 NISO Virtual Conference: Li... by National Informat... 1353 views
Sept 24 NISO Virtual Conference: Li... by National Informat... 1447 views
Sept 24 NISO Virtual Conference: Li... by National Informat... 1752 views
Sept 24 NISO Virtual Conference: Li... by National Informat... 1301 views

View on Slideshare

1 of 21 Ad

View on Slideshare

1 of 21 Ad