Downloaded 19 times
Uploaded byIan Foster
Rpi talk foster september 2011
The document discusses the challenges faced by small and medium-scale scientific projects in managing the increasing volume and complexity of data. It advocates for the adoption of Software as a Service (SaaS) to enhance research IT services, streamline data movement, and reduce costs through economies of scale. The use of cloud-based solutions, such as Globus Online, is proposed to improve efficiency in data management and facilitate collaboration among researchers.
More Related Content
Similar to Rpi talk foster september 2011
More from Ian Foster
Rpi talk foster september 2011
- 1. Accelerating data-intensive scienceby outsourcing the mundaneIan Foster
- 3. The data delugeMACHOet al.: 1 TBPalomar: 3 TB2MASS: 10 TBGALEX: 30 TBSloan: 40 TBPan-STARRS: 40,000 TB100,000 TBGenomic sequencing output x2 every 9 month>300 public centers1330molec. bio databases Nucleic Acids Research (96 in Jan 2001)2004: 36 TB2012: 2,300 TBClimate model intercomparisonproject (CMIP) of the IPCC
- 4. Big science hasachieved big successesOSG: 1.4M CPU-hours/day, >90 sites, >3000 users, >260 pubs in 2010LIGO: 1 PB data in last science run, distributed worldwideRobust production solutionsSubstantial teams and expenseSustained, multi-year effortApplication-specific solutions, built on common technologyESG: 1.2 PB climate datadelivered to 23,000 users; 600+ pubsAll build on NSF OCI (& DOE)-supported Globus Toolkit software
- 5. But small scienceis strugglingMore data, more complex dataAd-hoc solutionsInadequate software, hardwareData plan mandates
- 6. Medium-scale science strugglestoo!Blanco 4m on Cerro TololoImage credit: Roger Smith/NOAO/AURA/NSFDark Energy Survey receives 100,000 files each night in IllinoisThey transmit files to Texas for analysis … then move results back to IllinoisProcess must be reliable, routine, and efficientThe cyberinfrastructure team is not large
- 7. The challenge ofstaying competitive "Well, in our country," said Alice … "you'd generally get to somewhere else — if you run very fast for a long time, as we've been doing.” "A slow sort of country!" said the Queen. "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!"
- 8. Current approaches areunsustainableSmall laboratoriesPI, postdoc, technician, grad studentsEstimate 5,000 across US university communityAverage ill-spent/unmet need of 0.5 FTE/lab?Medium-scale projectsMultiple PIs, a few software engineersEstimate 500 across US university communityAverage ill-spent/unmet need of 3 FTE/project?Total 4000 FTE: at ~$100K/FTE => $400M/yr Plus computers, storage, opportunity costs, …
- 9. And don’t forgetadministrative costs42%of the time spent by an average PI on a federally funded research project was reported to be expended on administrative tasks related to that project rather than on research — Federal Demonstration Partnership faculty burden survey, 2007
- 10. You can runa company from a coffee shop
- 11. Because businesses outsourcetheir ITWeb presenceEmail (hosted Exchange)Calendar Telephony (hosted VOIP) Human resources and payroll Accounting Customer relationship mgmtSoftware as a Service(SaaS)
- 12. And often theirlarge-scale computing tooWeb presenceEmail (hosted Exchange)Calendar Telephony (hosted VOIP) Human resources and payroll Accounting Customer relationship mgmt Data analytics Content distributionSoftware as a Service(SaaS)Infrastructure as a Service(IaaS)
- 13. Let’s rethink howwe provide research ITAccelerate discovery and innovation worldwide by providing research IT as a serviceLeverage software-as-a-service toprovide millions of researchers with unprecedented access to powerful tools; enable a massive shortening of cycle times intime-consuming research processes; andreduce research IT costs dramatically via economies of scale
- 14. Time-consuming tasks inscienceRun experimentsCollect dataManage dataMove dataAcquire computersAnalyze dataRun simulationsCompare experiment with simulationSearch the literatureCommunicate with colleagues
- 15.
- 16. Find, configure, installrelevant software
- 17. Find, access, analyzerelevant data
- 18.
- 19.
- 20.
- 21. …Time-consuming tasks inscienceRun experimentsCollect dataManage dataMove dataAcquire computersAnalyze dataRun simulationsCompare experiment with simulationSearch the literatureCommunicate with colleagues
- 22.
- 23. Find, configure, installrelevant software
- 24. Find, access, analyzerelevant data
- 25.
- 26.
- 27.
- 28. …Data movement canbe surprisingly difficultBA
- 29. Data movement canbe surprisingly difficult Discover endpoints, determine available protocols, negotiate firewalls, configure software, manage space, determine required credentials, configure protocols, detect and respond to failures, determine expected performance, determine actual performance, identify diagnose and correct network misconfigurations, integrate with file systems, …It took 2 weeks and much help from many people to move 10 TB between California and Tennessee.(2007 BES report)BA
- 30. Globus Online’sSaaS/Web 2.0architectureCommand line interfacelsalcf#dtn:/scpalcf#dtn:/myfile \nersc#dtn:/myfileHTTP REST interfacePOST https://transfer.api.globusonline.org/ v0.10/transfer <transfer-doc>Web interfaceOpenIDOAuthShibboleth(Operate) Fire-and-forget data movementAutomatic fault recoveryHigh performanceNo client software installAcross multiple security domains(Hosted on) GridFTP serversFTP serversOther protocols:HTTP, WebDAV, SRM, …Globus Connecton local computers
- 31. Example application: UCsequencing facilityMac using Globus ConnectDelivery of data to customeriBi File ServerMount driveiBi general-purpose compute clusterSequencing-specific compute clusterSequencing instrument
- 32. Statistics and userfeedbackLaunched November 2010>1700 users registered>500 TB user data moved>30 million user files moved>150 endpoints registeredWidely used on TeraGrid/XSEDE; other centers & facilities; internationally>20x faster than SCPFaster than hand-tuned “Last time I needed to fetch 100,000 files from NERSC, a graduate student babysat the process for a month.”“I expected to spend four weeks writing code to manage my data transfers; with Globus Online, I was up and running in five minutes.”“Transferred 28 MB in 20 minutes instead of 61 hours. Makes these global climate simulations manageable.”
- 33. Moving 586 Terabytesin two weeks
- 34. Monitoring provides deepvisibility
Editor's Notes
- #2 New capabilities represent a tremendous opportunity for science.The challenge that I want to speak to is how we leverage these capabilities without computers and computation overwhelming the research community in terms of both human and financial resources.The solution, I will suggest, is to get computation out of the lab—to outsource it to third party providers. I will explain how this task can be achieved.
- #3 The need to deal with and benefit from large quantities of data is not a new concept: it has been noted in many policy reports, particularly in the US and UK, over the past several years.
- #4 But now the data deluge is now upon us. I use a few examples to highlight developments:-- Genome sequencing machines are doubling in output every nine months. This leaves the rather stately 18 month Moore’s Law doubling of computer performance in the shade.-- Astronomy, which only entered the digital era around 2000, projects 100,000 TB data from LSST by the end of the decade. [2MASS completed 2001; -- Simulation -- And not just volume, but also complexityTrends: Scale, complexity, distributed generation, …--------Source for genomic data: http://www.sciencemag.org/content/331/6018/728.short (“Output from next-generation sequencing (NGS) has grown from 10 Mb per day to 40 Gb per day on a single sequencer, and there are now 10 to 20 major sequencing labs worldwide that have each deployed more than 10 sequencers “)Source for mol bio dbs: http://nar.oxfordjournals.org/content/39/suppl_1/D1.full.pdf+htmlSource for climate change image: http://serc.carleton.edu/details/images/17685.html
- #7 Not just small labs—medium science too.E.g., Dark Energy Survey.
- #8 For many researchers, projects, and institutions, large data volumes are not an opportunity but a fundamental challenge to their competitiveness as researchers. How can they keep up?
- #9 200 universities * 250 faculty per university = 5,000Summary:-- Big projects can build sophisticated solutions to IT problems-- Small labs and collaborations have problems with both--They need solutions, not toolkits—ideally outsourced solutions
- #10 Need date
- #13 Of course, people also make effective use of IaaS, but only for more specialized tasks
- #14 More specifically, the opportunity is to apply a very modern technology—software as a service, or SaaS—to address a very modern problem, namely the enormous challenges inherent in translating revolutionary 21st century technologies into scientific advances. Midway’s SaaS approach will address these challenges, and both make powerful tools far more widely available, and reduce the cycle time associated with research and discovery.Achieve economies of scaleReduce cost per researcher dramaticallyAchieve positive returns to scaleMost academic solutions do NOT have PRTSMost industrial solutions DO have PRTS
- #15 So let’s look at that list again.I and my colleagues started an effort a little while ago aimed at applying SaaS to one of these tasks …
- #16 Example: small lab generates data at Texas Advanced Computing Center or the Advanced Photon Source. Needs to move it back to their lab.Or: Needs to move data from experimental facility (e.g., sequencing center or Dark Energy Survey) to computing facility for analysis.
- #17 Data movement is conceptually simple, but can be surprisingly difficult
- #18 Why? Discover endpoints, determine available protocols, negotiate firewalls, configure software, manage space, determine required credentials, configure protocols, detect and respond to failures, identify diagnose and correct network misconfigurations,…
- #19 •Reliable file transfer. –Easy “fire and forget” file transfers –Automatic fault recovery –High performance –Across multiple security domains•No IT required. –No client software installation –New features automatically available –Consolidated support and troubleshooting –Works with existing GridFTP servers –Globus Connect solves “last mile problem”
- #20 I’ll talk about integration with the Galaxy workflow system later …
- #21 Reduce costs.Improve performance.Enable new science.
- #25 What else do we need?
- #26 Add university logos?
- #30 Slide 33: Is the task of creating reusable workflows part of these 6 steps? Is publication and discovery of workflows/derived data products part of this as well? Is reproducible research part of it as well?
- #31 Researchers vote with their dollars
- #32 Before-- Lots of little labs-- Big science-- XSEDE After:lots of empowered SMLs, entrepreneurship in science, reproducible/reusable research etc