1.
SciMATE: A Novel MapReduce-Like
Framework for Multiple Scientific
Data Formats
Speaker: LIN Qian
http://www.comp.nus.edu.sg/~linqian

2.
Scientific data analysis today
• Increasingly data-intensive
– Volume approximately doubles each year
• Stored in certain specialized formats
– NetCDF, HDF5, ADIOS ...
• Popularity of MapReduce and its variants
– Free accessibility
– Easy programmability
– Good scalability
– Built-in fault tolerance
1

3.
NetCDF
• Network Common Data Form
2

4.
HDF5
• Hierarchical Data Format
3

5.
Scientific data analysis today (cont.)
• “Store-first-analyze-after”
– Reload data in another file system
E.g. load data from PVFS to HDFS
– Reload data in another data format
E.g. load NetCDF/HDF5 data to a specific
format
• Problems
– Long data migration/transformation time
– Stressing network and disks
4

6.
SciMATE
• In-situ scientific data analysis
– MapReduce with AlternaTE API
– Supporting NetCDF, HDF5 and flat-files
oNo data reloading!
– Transparent to app developers
• Optimized for
– Access strategies
– Access patterns
5

7.
System overview
6

8.
Scientific Data Processing Module
Runtime
System

9.
Integrating a new data format
• Data adaption layer is customizable
– Third-party adapter
– Open for extension but closed for
modification
• Have to implement the generic block
loader interface
– Partitioning function and auxiliary
functions
– Data access functions
8

10.
Data access strategies and patterns
• full_read()
– too expensive for reading small data
subsets
• partial_read()
– Strided pattern
o partial_read_by_block()
– Column pattern
o partial_read_by_column()
– Discrete point pattern
o partial_read_by_list()
9

11.
Access Pattern Optimization
• Strided pattern
– directly supported by API
• Discrete point pattern
– no optimization
• Column pattern
– fixed-size column read 1 2 3 4 5
– contiguous column read 1 2
10

12.
Evaluation
• System functionality and scalability
– 16 GB datasets
– Data processing times
ok-means, PCA, kNN
othread scalability, node scalability
– Data loading times
ok-means, PCA
onode scalability
• Partial read vs. Full read
• Fixed-size column read vs. Contiguous column
read
11

13.
Thread scalability

14.
Node scalability (data processing)

15.
Node scalability (data loading)

16.
Fixed-size column read vs. Contiguous column read
NetCDF HDF5

17.
Contiguous column read
NetCDF shows better column non-contiguity tolerance than HDF5.
16

18.
Conclusion and Future Work
• Conclusion
– Avoid bulk data transfers and vast data
transformation
– Provide a customizable data format
adaption API
– Support optimized read via access
strategies & patterns
• Future Work
– Compare with SciHadoop
17