The PARADIM Materials Innovation Platform has launched a revolutionary, streaming-data-based approach to the design and creation of new interface materials – new materials with the unprecedented properties needed to realize exciting technologies like quantum computing. PARADIM is a state-of-the-art scientific facility at Cornell, Johns Hopkins, and Clark Atlanta Universities that produces a firehose of complex and highly varied data. To build our data layer we’ve implemented Apache Kafka with newly developed, end-to-end encryption to connect our instruments, users, and world-class experts. Data reduction, analysis and management connect seamlessly to real-time ML tools in an integrated materials design workflow. At PARADIM, this flexible, scalable, and fault-tolerant data streaming is creating a new paradigm for collaborative science in the age of materials Big Data.

1. StreamingTowards Our Quantum Future
streaming data for materials science research on
quantum materials in the PARADIM Materials
Innovation Platform
David Elbert1, Nicholas S. Carey2,Tyrel McQueen3
Johns Hopkins University
1. Hopkins Extreme Materials Institute
2. Department of Computer Sciences
3. Departments of Chemistry; Physics and Astronomy; and Materials Science and Engineering

2. Materials Science Data
De Carlo et al., 2012
• Higher resolution
• Shorter time scales
• Higher dimensionality
• Dynamic experiments
• Larger simulations
• Tighter processing control
Shibuta et al., 2017

3. PARADIM
User Facility:
• Crystal Growth Facility (Hopkins)
• PPMS and Characterization (Hopkins)
• Thin Film Deposition (Cornell)
• Electron Microscopy (Cornell)
• Theory/Modeling (Cornell/Clark
Atlanta)
Data Challenges:
• User access/Security/Privacy
• LargeVolume/Dispersed Activity
• Scalable visualization/analysis
• Education for broad user base
• Changing/Evolving infrastructure
• Time value of data

4. Many Moving Parts
• Equipment
• People
• Ideas

5. Key KafkaTenets
• Persists Data
• Scalable
• Flexible
• Modular
• Plays well with others
• Cost

6. 1. Cornell Electron Microscopy

7. 1. Cornell Electron Microscopy

8. 1. Cornell Electron Microscopy

9. Sending arbitrarily large files
• Asynchronous upload
• Hash each chunk
• Chunks as messages
• Construct info message
• Header metadata to DB

10. 2. Hopkins Optical Floating Zone Furnace
Boron Carbide:

11. 2. Hopkins Floating Zone Furnace

12. 2. Hopkins Floating Zone Furnace

13. Instance Segmentation
Mask R-CNN
He et al, 2017

14. Realtime Streaming Data Deployment

15. Data RegainsValue OverTime

16. 3. Kafka Crypto
• Data confidentiality
• Send before peers known
• Forward secrecy
• Asymmetric primitives
• Post-Quantum ready

17. Conclusions
• Streaming opens new vistas
• Scalability, flexibility, ease
• Evolving data lifecycle