Modern scientific experiments generate vast volumes of data which are hard to keep track of. Because of that scientists find it difficult to reuse these data. We address this problem by implementing a scientific data cataloging system for efficient management of scientific data. It is developed using Solr as the back end. Because of it our system can outperforms MySQL relational database solution by 94% in full text search queries, 99% in sub string queries, 99% in prefix queries and 91% in suffix queries for 100,000 records. Another main advantage in our system is that it can be extended to suit any type of scientific data cataloging requirement.

1. Schema–Independent Scientific
Data Cataloging Framework
Supun Nakandala, Sachith Dhanushka Withana, Dinu
Kumarasiri, Hirantha Jayawardena and H.M.N. Dilum
Bandara
(Department of Computer Science and Engineering, University of
Moratuwa, Sri Lanka)
Srinath Perera
(Wso2 Inc., Colombo, Sri Lanka)
Suresh Marru, Sudhakar Pamidighantam
Indiana University, Bloomington, USA
1

2. Scientist
Problem
Scientific Data
✧ Vast Volume
✧ Hard to …
❑ Search
❑ Reuse
❑ Share findings
2

3. GridChem Usecase
• Gaussian 9 experiments generate vast amount
of data in two forms
• Output file (*.out)
• Check point file (*.chk)
• Provide efficient searching among these data
3

4. Why we need a new one ??
Existing Solutions
● Tightly coupled
● Inflexible querying
● Static schemas
● Eg:-
● MCS
● MCAT
● MyLEAD
4
Our Solution
● Generalizable framework
● Flexible querying
o Wild card queries
o Full text queries
o Substring queries
o Fielded queries
● Static schema + dynamic fields

5. m
5
High-level Architecture

6. Folder Structure
6

7. What is new in our solution?
• Pluggable metadata extraction logic
• Extensible data product generation monitors
• Use of NoSQL database (Apache Solr)
• Ability to dynamically add metadata fields
7

8. Performance Test
8
• MySQL vs Solr
• Data Insert Performance
• Query Performance
• Exact match queries
• Range queries
• Full text queries
• Prefix match queries
• Suffix match queries
• Wildcard queries
• Substring queries
Solr resolves more complex queries 91% - 99% faster than a
MySQL-based implementation.

9. 9

10. 10

11. 11

12. Summary
• What we did: A schema-independent
scientific data catalog with pluggable parser
logic and Solr backend
• Future work: Airavata integration and
provenance aware execution
12

13. Thank You …
1
3