Download to read offline
![Understanding Incentives is Critical
50
Meeting Award
Requirements
Smoothing
Dislocations
Increasing
Impact
• Increase paper citations1
• Add dataset citation capabilities
• [Distance] Enable simple sharing among
collaborators (near and far)
• [Personnel] Ease transitions between students
• [Format] Lessen need for ad hoc resource sharing
(e.g. via group websites)
• Simplify DMP compliance
1 Citation increase 30 (10.7717/peerj.175) - 60% (10.1371/journal.pone.0000308) [caveat bio research]](https://image.slidesharecdn.com/20160922-mdf-nist-webinar-f-160927144535/75/20160922-Materials-Data-Facility-TMS-Webinar-42-2048.jpg)
![Lessons Learned
51
• The demand is there from researchers and
institutions
• Lots of cross-over with centers and projects
§ (NIST) CHiMaD
§ (DOE) ElectroCat, MICCoM, JCESR, PRISMS, Argonne IT, Integrated Imaging Institute
§ (NSF) T2C2 [DIBBS], AMI-CFP (PIRE), HV/TMS (I/UCRC), BD Hubs, IMaD BD Spoke*
• Data Heterogeneity is a challenge
§ Metadata is the major sticking point
• Friction points
§ Need more flexible data objects e.g. {“temperature”:100, “unit”:“K”}
§ Need file or directory based metadata
§ Immutable datasets alone is not enough à Versioning
§ Data gathering in retrospect
§ Schema generation and interoperability
§ Working with and following developments at NIST, RDA, Citrine et al.
§ Differing institutional approval processes
§ Lack of programmatic interface (planned).
• Support for data interactivity and visualization
• Smart versioning for large file-based datasets](https://image.slidesharecdn.com/20160922-mdf-nist-webinar-f-160927144535/75/20160922-Materials-Data-Facility-TMS-Webinar-43-2048.jpg)
Uploaded byBen Blaiszik
20160922 Materials Data Facility TMS Webinar
The document outlines the development and features of the Materials Data Facility (MDF), which aims to simplify the publication, discovery, and use of materials datasets through a series of APIs and cloud-hosted services. It discusses the challenges of materials data publication, including the need for unique identifiers, metadata schemas, and automated curation workflows. Furthermore, the MDF is positioned as a valuable resource for researchers, promoting data accessibility and collaboration in the materials science community.
More Related Content
Similar to 20160922 Materials Data Facility TMS Webinar
20160922 Materials Data Facility TMS Webinar
- 1. Ben Blaiszik (blaiszik@uchicago.edu), KyleChard, Rachana Ananthakrishnan Michael Ondrejcek, Kenton McHenry PIs: Ian Foster (foster@uchicago.edu), Steven Tuecke, John Towns materialsdatafacility.org globus.org Materials Data Facility - Data Services to Advance Materials Science Research
- 2. 2 http://dx.doi.org/10.1007/s11837-016-2001-3 MDF Article inJOM (August Issue)
- 3.
- 4. 4 Outline APIs • Overview § MDFOverview § Globus quick introduction • MDF Data Publication Service § Key MDF data pub service features § Publication walk-through • General Observations and Future Outlook
- 5. What is MDF? 5 Weare developing production services to make it more simple for materials datasets and resources to be ... Published Identified Described Curated Verifiable Accessible Preserved Discovered Searched Browsed Shared Recommended Accessed and SRD Publishable Results Published Results Resource Data Ref Data Derived Data Working Data * Figure adapted from Warren et al.
- 6. Data Service Infrastructure 6 PublicationDiscovery Compute for data interaction and viz Resource Registration APIs + + + - Initial Foci
- 7. 7 Publication APIs • Identify datasetswith persistent identifiers (e.g. DOI) • Describe datasets with appropriate metadata and provenance • Verify dataset contents over time • Preserve critical datasets in a state that increases transparency, replicability, and helps encourage reuse
- 8. 8 Discovery • Search andquery datasets in modern ways – e.g. via search against indexed metadata and harvested file contents rather than remembering opaque file paths Future... Spotlight for all data you have access to regardless of location Under Development
- 9. 9 Discovery Under Development • SaaScloud-hosted solution • Logical metadata repository to index many external sources • Flexible queries (boosting, full text, partial matches, etc.) • Search results are limited by ACLs
- 10. 10 Discovery Under Development • AllMDF-published datasets will be indexed • May use common schemas (Datacite, Dublin Core etc.) or domain specific • Globus endpoint contents may be indexed (owner enabled) • Index has the flexibility of no required schema • Built on Elasticsearch for proven scalability and speed, hosted on scalable AWS resources
- 11.
- 12.
- 13. Globus Platform-as-a-Service (PaaS) 14 Identity management User groups Data transfer Data sharing •Share directly from your storage device (laptop or cluster) • File and directory-level ACLs • Manage user group creation and administration flows • Share data with user groups • High-performance data transfer from a web browser • Optimize transfer settings and verify transfer integrity • Add your laptop to the Globus cloud with Globus Connect Personal • create and manage a unique identity linked to external identities for authentication Publication Discovery
- 14. REST APIs, Clients,and Docs 15 • New version of core services released in Feb. • New Python SDK available § https://github.com/globusonline/globus-sdk-python • Jupyter Notebook Examples § https://github.com/globus/globus-jupyter-notebooks • Sample Data Portal § https://github.com/globus/globus-sample-data-portal • (alpha) MDF Data Publication Service API
- 15. Globus Background 16 B Globus movesthe data for you secure endpoint, e.g. laptop You submit a transfer request Globus notifies you once the transfer is complete secure endpoint, e.g. midway transfer A Endpoint • E.g. laptop or server running a Globus client (e.g. Dropbox client) • Enables advanced file transfer and sharing • Currently GridFTP, future GridFTP + HTTP Some Key Features • REST API for automation and interoperability • Web UI for convenience • Optimizes and verifies transfers • Handles auto-restarts • Battle tested with big data
- 16. Globus Web UI 17 Endpoint •E.g. laptop or server running a Globus client (e.g. Dropbox client) • Enables advanced file transfer and sharing • Currently GridFTP, future GridFTP + HTTP Some Key Features • REST API for automation and interoperability • Web UI for convenience • Optimizes and verifies transfers • Handles auto-restarts • Battle tested with big data
- 17.
- 18. 20 Materials Data Publication/Discoveryis Often a Challenge Data Collection Data Storage and Process Publication
- 19. 21 Materials Data Publication/Discoveryis Often a Challenge Data Collection ? ? ? Networked storage, sometimes many TB Unique identifier data for search/cite Custom metadata descriptions Data curation workflow Automation capabilities Data Storage and Process Publication Want to Discover / Use Want to Publish Don’t put under desk! Needed to close the loop
- 20. 22 Data Collection ? ? ? Need storage,sometimes many TB Need to uniquely identify data for search/cite Need custom metadata descriptions Need a data curation workflow Need automation capabilities Data Storage and Process Publication Want to Discover / Use Want to Publish Materials Data Publication/Discovery is Often a Challenge Don’t put under desk!
- 21. Collection Model 23 • Collectionsmight be a research group or a research topic... • Collections have specified § Mapping to storage endpoint § Currently handled as automatically created shared endpoints § Metadata schemas § Access control policies § Licenses § Curation workflows • Collections contain § Datasets § Data § Metadata • Metadata Persistence § Metadata log file with dataset § Metadata replicated in search index
- 22. Hybrid Distributed Model 24 Petrel@Argonne 1.7 PB BlueWaters Condo @UIUC 100 TB EP 1 EP 2 EP 3 Campus RDS DOE Cloud Metadata Index And Tools Centralized resource Globus endpoint NSF (XSEDE) ElectroCat EP
- 23. Publish Large Datasets 25 •Distributed data model leverages Globus production capabilities for file transfer (i.e. dataset assembly), user authentication, and access control groups • 100s of TB of reliable storage @ NCSA, and more storage at Argonne § Globus endpoint at ncsa#mdf on Nebula § Expandable to many PBs as necessary § Automated tape backup for reliability (in progress) • Researchers can optionally use your own local or institutional storage
- 24. Uniquely Identify Datasets 26 •Associate a unique identifier with a dataset § DOI, Handle • Improve dataset discovery and citability § Aligning incentives and understanding the culture will be critical to driving adoption DatasetDownloads Time • Your work has been cited 153 times in the last year • Researchers from 30 institutions have downloaded your datasets Future...
- 25. Share Data withFlexible ACLs 27 • Share data publicly, with a set of users, or keep data private Leverage Curation Workflows • Collection administrators can specify the level of curation workflow required for a given collection e.g. § No curation § Curation of metadata only § Curation of metadata and files
- 26. Customize Metadata 28 • Builda custom metadata schema for your specific research data • Re-use existing metadata schemas • Working in conjunction with NIST researchers to define these schemas • Can we build a system that allows schema: § Inheritance § E.g. a schema “polymers” might inherit and expand upon the “base material” of NIST § Versioning § E.g. Understand contextually how to map fields between versions § Dependence § E.g. Allows the ability to build consensus around schemas Future...
- 27.
- 28. Example Use Case 30 PublishingBig, Remote Data Collected multi TB of data at a light source Bundle the data with metadata and provenance Want a citable DOI to share the raw and derived data with the community Want their data to be discoverable by free text search and custom metadata
- 29.
- 30. MDF Collections 32 Recall: PoliciesSet at the Collection Level • Required metadata, schemas • Data storage location • Metadata curation policies
- 31. MDF Metadata Entry 33 •Scientist or representative describes the data they are submitting • For this collection Dublin Core and a custom metadata template are required
- 32. MDF Custom Metadata 34 •Scientist or representative describes the data they are submitting • For this collection Dublin Core and a custom metadata template are required
- 33. Dataset Assembly 35 • Sharedendpoint is auto-created on collection-specified data store • Scientist transfers dataset files to a unique publish endpoint • Dataset may be assembled over any period of time • When submission is finished, dataset will be rendered immutable via checksum (e.g. NU) (e.g. UIUC Nebula)
- 34. Dataset Assembly 36 • Sharedendpoint is auto-created on collection-specified data store • Scientist transfers dataset files to a unique publish endpoint • Dataset may be assembled over any period of time • When submission is finished, dataset will be rendered immutable via checksum (e.g. NU) (e.g. UIUC Nebula)
- 35. Dataset Curation (Optional) 37 •Optionally specified in collection configuration • Can be approved or rejected (i.e. sent back to the submitter)
- 36. Mint a PermanentIdentifier 38 Can be DOI or Handle
- 37.
- 38.
- 39.
- 40. 48 Publication Year 1 Milestones APIs •Opened to the public in March 2016 • Provisioned reliable storage to support researchers sharing open materials data (~200 TB) • MDF data volume approaching ~ 6 TB of materials data • Started building deep relationships with many of the key materials data generating groups and communities • Ingested dataset > 1 TB in size • Ingested dataset > 1.5M files
- 41. Integration with theCommunity is Key 49 Materials Project OQMD Citrination Materials Commons Other Facilities (APS, SNS, NSLS, …), Institutional Repositories, Publishers! Metadata Publishing MetadataMD, Pub., Compute Metadata Publishing NCSA-PIREHV/TMSMBDH
- 42. Understanding Incentives isCritical 50 Meeting Award Requirements Smoothing Dislocations Increasing Impact • Increase paper citations1 • Add dataset citation capabilities • [Distance] Enable simple sharing among collaborators (near and far) • [Personnel] Ease transitions between students • [Format] Lessen need for ad hoc resource sharing (e.g. via group websites) • Simplify DMP compliance 1 Citation increase 30 (10.7717/peerj.175) - 60% (10.1371/journal.pone.0000308) [caveat bio research]
- 43. Lessons Learned 51 • Thedemand is there from researchers and institutions • Lots of cross-over with centers and projects § (NIST) CHiMaD § (DOE) ElectroCat, MICCoM, JCESR, PRISMS, Argonne IT, Integrated Imaging Institute § (NSF) T2C2 [DIBBS], AMI-CFP (PIRE), HV/TMS (I/UCRC), BD Hubs, IMaD BD Spoke* • Data Heterogeneity is a challenge § Metadata is the major sticking point • Friction points § Need more flexible data objects e.g. {“temperature”:100, “unit”:“K”} § Need file or directory based metadata § Immutable datasets alone is not enough à Versioning § Data gathering in retrospect § Schema generation and interoperability § Working with and following developments at NIST, RDA, Citrine et al. § Differing institutional approval processes § Lack of programmatic interface (planned). • Support for data interactivity and visualization • Smart versioning for large file-based datasets
- 44. Wider Data Community 52 •Curated and described datasets • Well-posed problems • Community to share analyses • Challenges to start “sprints” • Great APIs and clients • Examples to get started • Hundreds of video tutorials Materials ProjectOQMD Citrination Materials Commons • Less inherently intuitive problems • Sometimes need advanced compute capabilities • Often many TB
- 45. 53 • Continuous integration,QA, and testing • Containerized solutions, microservice architecture, abstracting software from hardware • Automation • Internet of Things (IoT) – connect everything • Machine Learning / AI • Natural Language Processing (Siri, chatbots or “slack”bots, etc.) • Search rules the world – ok this was 20 years ago… What are the analogs and applications in the materials community? Materials ProjectOQMD Citrination Materials Commons • Less inherently intuitive problems • Sometimes need advanced compute capabilities • Often many TB Broader Trends
- 46. 54 Experimentation Ahead No teamcommitments here! Open source opportunities, contact: blaiszik@uchicago.edu
- 47. Use Case: ScenarioGenerator-Consumer 55 • Data generator § Generates data periodically (perhaps from an instrument) § Pushes data to a public channel § Schema is validated before inclusion in channel stream • Data consumer § Polls channel periodically § Wants to pull datasets by property Dataset Channel MDF-composites Data Generator Data Consumer DatasetDatasetDataset DatasetDatasetcreate q: result q
- 48.
- 49. Aggregate, Perform ML 58 •Combine cloud-published dataset, scikitlearn, pandas to predict steel fatigue and “reproduce” data from journal publication
- 50. Aggregate, Perform ML,Visualize 59 • Combine cloud-published dataset, scikitlearn, pandas to predict steel fatigue and validate journal publication
- 51. What’s Currently Available? 60 •Web interface to support data publication (public- facing APIs coming soon) • 100s of TB of storage at NCSA (scalable to many PB) more at Argonne (1.7 PB total on Petrel – not all for materials…) • Help with developing metadata schemas to describe your research datasets MDF Tutorial on Github https://github.com/blaiszik/materials-data-facility-training
- 52. What are welooking for? 61 • Early adopters, willing to get their hands dirty with the service and give honest feedback • Key integration points where metadata is picked up automatically! • Key datasets and resources of all sizes, shapes, raw or derived, that might help us understand the process better
- 53. Thanks to OurSponsors! 62 U . S . D E P A RT M E N T O F ENERGY
- 54. Publication REST APIsDiscovery • Identify datasets with persistent identifiers (e.g. DOI) • Describe datasets with appropriate metadata and provenance • Verify dataset contents over time • Handle big (and small) data: We have already ingested datasets with > 1.5M files and > 1TB in size • Search and query datasets in modern ways • Index metadata and harvest file contents • Simple user interfaces (i.e., after Google and Amazon) Opened to external users in Mar. 2016 ~ 6 TB of data published Materialsdatafacility .org