This document discusses research automation and data-driven discovery. It notes that data volumes are growing much faster than computational power, creating a productivity crisis in research. However, most labs have limited resources to handle these large data volumes. The document proposes applying lessons from industry to create cloud-based science services with standardized APIs that can automate and outsource common tasks like data transfer, sharing, publishing, and searching. This would help scientists focus on their core research instead of computational infrastructure. Examples of existing services from Argonne National Lab and the University of Chicago Globus project are provided. The goal is to establish robust, scalable, and persistent cloud platforms to help address the challenges of data-driven scientific discovery.

1. Research Automation
for Data-Driven Discovery
Ian Foster
Argonne National Laboratory &
The University of Chicago
foster@anl.gov

2. A productivity crisis in research
Data volumes are growing
much faster than Moore’s law …
(10,000x more over 6 years for
genome data)
Kahn, Science, 331
(6018): 728-729
But most labs
have extremely
limited resources
Heidorn: NSF
grants in 2007
< $350,000
80% of awards
50% of grant $$

3. "Well, in our country," said Alice …
"you'd generally get to somewhere else
— if you run very fast for a long time,
as we've been doing.”
"A slow sort of country!" said the
Queen. "Now, here, you see, it
takes all the running you can do,
to keep in the same place. If you
want to get somewhere else, you
must run at least twice as fast as that!"
The challenge of staying competitive

4. 4https://bit.ly/2l4gfgu
How industry handles complexity

5. cloud4scieng.org
Industry software builds on powerful platform services

6. Cloud platforms have transformed how software is
developed and delivered
6
Can we do the same for science?
• Identify cross-cutting capabilities required by many groups
• Define simple REST APIs for accessing those capabilities
• Operate high-quality, scalable, secure, performant cloud-hosted
implementations
• Ensure persistence and evolution over time
In so doing, enable many scientists and tool developers to automate
and outsource tasks that are not central to their core mission: thus
reduce costs, increase quality, promote interoperability

7. What capabilities?
7
• Auth: Manage identities, authentication, and authorization
• Transfer: Manage movement of files from A to B
• Sharing: Manage who can access data at a location
• Publish: Preserve, identify, describe, curate
• Search: Index and search data
• Identifiers: Assign identifiers to collections of files
• Automate: Organize sets of activities
• Learn: Discover, train, run machine learning models
• …

8. globus.org
Science services
operated by UChicago for researchers worldwide

9. Monitor transfer
Monitor activitiesManage data

10. Automate and outsource with
REST APIs and Python SDK

11. Automate and
outsource with
REST APIs and
Python SDK
11
UK
NIST
NSF
NSF
NSF
DOE
NSF
Canada

12. Automate and outsource:
Publication and discovery
Move to permanent location
(or publish in place)
Compute and record checksums
Obtain and record metadata
Assign persistent identifier
Index for discovery
1212
Data Publication
Indexing
materialsdatafacility.org
2 petabytes
100 Gbps
Globus APIs

13. Automate and outsource:
Publication and discovery
1313
Programmatic access (Python, Jupyter)
Web browse and search
Data Publication
Indexing
materialsdatafacility.org
2 petabytes
100 Gbps
Globus APIs

14. Example: NCAR’s Research Data Archive
Globus used for
• Single sign on via
streamlined account
provisioning
• Data sharing
• Data downloads

15. 15

16. Beyond transfer
(Experimental)

17. Cloud platforms have transformed how software is
developed and delivered
17
We can do the same for science
• Identify cross-cutting capabilities required by many groups
• Define simple REST APIs for accessing those capabilities
• Operate high-quality, scalable, secure, performant cloud-hosted
implementations
• Ensure persistence and evolution over time
In so doing, enable many scientists and tool developers to automate
and outsource tasks that are not central to their core mission, to
reduce costs, increase quality, promote interoperability

18. We have identified some needed capabilities
18
• Auth: Manage identities, authentication, authorization
• Transfer: Manage movement of files from A to B
• Sharing: Manage who can access data at a location
• Publish: Preserve, identify, describe, curate
• Search: Index and search data
• Identifiers: Assign identifiers to collections of files
• Automate: Organize sets of activities
• Learn: Discover, train, run machine learning models
• …
Established
12,000 endpoints
100,000+ users
New
100s of users
Experimental
10s of users
globus.org — Ian Foster — foster@anl.gov