The document discusses the challenges and opportunities in research data management, emphasizing the need for scalable, cloud-based solutions to handle the increasing volume and complexity of data across various scientific fields. It highlights the inefficiencies of current practices and proposes a shift towards Research IT as a Service to enhance accessibility, streamline processes, and reduce costs. Through leveraging Software-as-a-Service, the goal is to democratize access to powerful research tools, ultimately accelerating discovery and innovation.

1. So long, computer overlordsHow Cloud (and Grid) can liberate research IT – and transform discoveryIan Foster

4. The data delugeMACHO et al.: 1 TBPalomar: 3 TB2MASS: 10 TBGALEX: 30 TBSloan: 40 TBPan-STARRS: 40,000 TB100,000 TBGenomic sequencing output x2 every 9 month>300 public centers1330molec. bio databases Nucleic Acids Research (96 in Jan 2001)2004: 36 TB2012: 2,300 TBClimate model intercomparisonproject (CMIP) of the IPCC

5. Big science has achieved big successesOSG: 1.4M CPU-hours/day, >90 sites, >3000 users, >260 pubs in 2010LIGO: 1 PB data in last science run, distributed worldwideRobust production solutionsSubstantial teams and expenseSustained, multi-year effortApplication-specific solutions, built on common technologyESG: 1.2 PB climate datadelivered to 23,000 users; 600+ pubsAll build on NSF OCI (& DOE)-supported Globus Toolkit software

6. But small science is strugglingMore data, more complex dataAd-hoc solutionsInadequate software, hardwareData plan mandates

7. Medium-scale science struggles too!Blanco 4m on Cerro TololoImage credit: Roger Smith/NOAO/AURA/NSFDark Energy Survey receives 100,000 files each night in IllinoisThey transmit files to Texas for analysis … then move results back to IllinoisProcess must be reliable, routine, and efficientThe cyberinfrastructure team is not large

8. The challenge of staying competitive "Well, in our country," said Alice … "you'd generally get to somewhere else — if you run very fast for a long time, as we've been doing.” "A slow sort of country!" said the Queen. "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!"

9. Current approaches are unsustainableSmall laboratoriesPI, postdoc, technician, grad studentsEstimate 5,000 across US university communityAverage ill-spent/unmet need of 0.5 FTE/lab?Medium-scale projectsMultiple PIs, a few software engineersEstimate 500 across US university communityAverage ill-spent/unmet need of 3 FTE/project?Total 4000 FTE: at ~$100K/FTE => $400M/yr Plus computers, storage, opportunity costs, …

10. And don’t forget administrative costs42%of the time spent by an average PI on a federally funded research project was reported to be expended on administrative tasks related to that project rather than on research — Federal Demonstration Partnership faculty burden survey, 2007

11. You can run a company from a coffee shop

12. Because businesses outsource their ITWeb presenceEmail (hosted Exchange)Calendar Telephony (hosted VOIP) Human resources and payroll Accounting Customer relationship mgmtSoftware as a Service(SaaS)

13. And often their large-scale computing tooWeb presenceEmail (hosted Exchange)Calendar Telephony (hosted VOIP) Human resources and payroll Accounting Customer relationship mgmt Data analytics Content distributionSoftware as a Service(SaaS)Infrastructure as a Service(IaaS)

14. Let’s rethink how we provide research ITAccelerate discovery and innovation worldwide by providing research IT as a serviceLeverage software-as-a-service toprovide millions of researchers with unprecedented access to powerful tools; enable a massive shortening of cycle times intime-consuming research processes; andreduce research IT costs dramatically via economies of scaleso long, computer overlords

15. Time-consuming tasks in scienceRun experimentsCollect dataManage dataMove dataAcquire computersAnalyze dataRun simulationsCompare experiment with simulationSearch the literatureCommunicate with colleagues

16. Publish papers

17. Find, configure, install relevant software

18. Find, access, analyze relevant data

19. Order supplies

20. Write proposals

21. Write reports

22. …Time-consuming tasks in scienceRun experimentsCollect dataManage dataMove dataAcquire computersAnalyze dataRun simulationsCompare experiment with simulationSearch the literatureCommunicate with colleagues

23. Publish papers

24. Find, configure, install relevant software

25. Find, access, analyze relevant data

26. Order supplies

27. Write proposals

28. Write reports

29. …Data movement can be surprisingly difficultBA

30. Data movement can be surprisingly difficult Discover endpoints, determine available protocols, negotiate firewalls, configure software, manage space, determine required credentials, configure protocols, detect and respond to failures, determine expected performance, determine actual performance, identify diagnose and correct network misconfigurations, integrate with file systems, …It took 2 weeks and much help from many people to move 10 TB between California and Tennessee.(2007 BES report)BA

31. Globus Online’sSaaS/Web 2.0 architectureCommand line interfacelsalcf#dtn:/scpalcf#dtn:/myfile \nersc#dtn:/myfileHTTP REST interfacePOST https://transfer.api.globusonline.org/ v0.10/transfer <transfer-doc>Web interface(Operate) Fire-and-forget data movementAutomatic fault recoveryHigh performanceNo client software installAcross multiple security domains(Hosted on) GridFTP serversFTP serversOther protocols:HTTP, WebDAV, SRM, …Globus Connecton local computers

32. Example application: UC sequencing facilityMac using Globus ConnectDelivery of data to customeriBi File ServerMount driveiBi general-purpose compute clusterSequencing-specific compute clusterSequencing instrument

33. Statistics and user feedbackLaunched November 2010>1400 users registered>350 TB user data moved>28 million user files moved>140 endpoints registeredWidely used on TeraGrid/XSEDE; other centers & facilities; internationally>20x faster than SCPFaster than hand-tuned “Last time I needed to fetch 100,000 files from NERSC, a graduate student babysat the process for a month.”“I expected to spend four weeks writing code to manage my data transfers; with Globus Online, I was up and running in five minutes.”“Globus Online’s speed has us planning experiments that we would never have considered previously.”

34. Moving 586 Terabytes in two weeks

35. Monitoring provides deep visibility

36. 20 Terabytes in less than one dayTerabyte 20 Gigabyes in more than two daysGigabyte Megabyte Kilobyte

37. Common research data management stepsDark Energy SurveyGalaxy genomicsLIGO observatorySBGrid structural biology consortiumNCAR climate data applicationsLand use change; economics

38. We have choices of where to computeCampus systemsFirst target for many researchersXSEDE supercomputers220,000 cores, peer-reviewed awardsOptimized for scientific computingOpen Science Grid60,000 cores; high throughputCommercial cloud providersInstant access for small tasksExpensive for big projectsUsers insist that they need everything connected

39. Towards “research IT as a service”

40. Research data management as a serviceGO-UserCredentials and other profile informationGO-TransferData movementGO-TeamGroup membershipGO-CollaborateConnect to collaborative tools: Jira, Confluence, …GO-StoreAccess to campus, cloud, XSEDE storageGO-CatalogOn-demand metadata catalogsGO-ComputeAccess to computersGO-GalaxyShare, create, run workflowsTodayPrototypeFall

41. SaaS services in action: The XSEDE visionXUAS

42. Data analysis as a service: Early stepsSecurely and reliably:Assemble codeFind computersDeploy codeRun programAccess dataStore dataRecord workflowReuse workflow[7, 8][1, 2]We have built such systems for biological, environmental,and economics researchersVM imageApp codeWorkflowGalaxyCondor[3, 4][5, 6]Data store

43. SaaS economics: A quick tutorialLower per-user cost (x10?) via aggregation onto common infrastructure$400M/yr $40M/yr?Initial “cost trough” due to fixed costsPer-user revenue permits positive return to scale Further reduce per-user cost over time$0TimeX10 reduction in per-user cost: $50K  $5K/yr per lab $300K  $30K/yr per project

44. A national cyberinfrastructure strategy?To providemore capability formore people at less cost …Create infrastructure Robust and universalEconomies of scalePositive returns to scaleVia the creative use ofAggregation (“cloud”)Federation (“grid”)Small and medium laboratories and projectsPLLLLLLLLLPPPPLLLLLLLLLLLLLLLLLLaaSResearch data management Collaboration, computationResearch administration

45. AcknowledgmentsColleagues at UChicago and ArgonneSteve Tuecke, Ravi Madduri, Kyle Chard, Tanu Malik, Michael Russell, Paul Dave, Stuart Martin, Dan Katz, and many othersColleagues at other institutionsCarl Kesselman, MironLivny, John Towns, and othersNSF OCI, MPS, and SBE; DOE ASCR; and NIH for support

46. For more informationFoster, I. Globus Online: Accelerating and democratizing science through cloud-based services. IEEE Internet Computing(May/June):70-73, 2011.Allen, B., Bresnahan, J., Childers, L., Foster, I., Kandaswamy, G., Kettimuthu, R., Kordas, J., Link, M., Martin, S., Pickett, K. and Tuecke, S. Globus Online: Radical Simplification of Data Movement via SaaS. Communications of the ACM, 2011.

47. Thank you!foster@uchicago.edu