Download as PDF, PPTX
Uploaded byKirill Osipov
Science cloud foster june 2013
This document discusses building a "discovery cloud" to accelerate scientific discovery through on-demand computing. It proposes identifying time-consuming research activities that can be automated and outsourced as software-as-a-service. This would achieve economies of scale through leveraging infrastructure-as-a-service. The goal is to create a great user experience for scientific data management similar to consumer services like Dropbox. It also discusses integrating services like Globus for data management and Galaxy for analysis to provide flexible and scalable genomics analysis.
More Related Content
Similar to Science cloud foster june 2013
Science cloud foster june 2013
- 1. computationinstitute.org Science as aservice How on-demand computing can accelerate discovery Ian Foster foster@anl.gov
- 2. computationinstitute.org A time ofdisruptive change As evidenced by cost per human genome
- 3. computationinstitute.org A time ofdisruptive change As evidenced by cost per human genome
- 4. computationinstitute.org But most labshave extremely limited resources Heidorn: NSF grants in 2007 < $350,000 80% of awards 50% of grant $$ $1,000,000 $100,000 $10,000 $1,000 2000 4000 6000 8000
- 5. computationinstitute.org Automation is required toapply more sophisticated methods to far more data
- 6. computationinstitute.org Automation is required toapply more sophisticated methods to far more data Outsourcing is needed to achieve economies of scale in the use of automated methods
- 7. computationinstitute.org Building a discoverycloud • Identify time-consuming activities amenable to automation and outsourcing • Implement as high-quality, low-touch SaaS • Leverage IaaS for reliability, economies of scale • Extract common elements as research automation platform Bonus question: Sustainability Software as a service Platform as a service Infrastructure as a service
- 8. computationinstitute.org Where does timego in research? 42%!!
- 9. computationinstitute.org We aspire (initially)to create a great user experience for research data management What would a “dropbox for science” look like?
- 10. computationinstitute.org • Collect • Move •Sync • Share • Analyze • Annotate • Publish • Search • Backup • Archive BIG DATA
- 11. computationinstitute.org Registry Staging Store Ingest Store Analysis Store Community Store Archive Mirror Ingest Store Analysis Store Community Store Archive Mirror Registry Quota exceeded ! Expired credentials ! Network failed.Retry. ! Permission denied ! It should be trivial to Collect, Move, Sync, Share, Analyze, Annotate, Publish, Search, Backup, & Archive BIG DATA … but in reality it’s often very challenging
- 12. computationinstitute.org • Collect • Move •Sync • Share • Analyze • Annotate • Publish • Search • Backup • Archive BIG DATA
- 13. computationinstitute.org • Collect • Move •Sync • Share • Analyze • Annotate • Publish • Search • Backup • Archive • Collect • Move • Sync • Share Capabilities delivered using Software-as-Service (SaaS) model
- 14.
- 15. computationinstitute.org Data Source User A selects file(s)to share; selects user/group, sets share permissions 1 Globus Online tracks shared files; no need to move files to cloud storage! 2 User B logs in to Globus Online and accesses shared file 3
- 16. computationinstitute.org Extreme ease ofuse • InCommon, Oauth, OpenID, X.509, … • Credential management • Group definition and management • Transfer management and optimization • Reliability via transfer retries • Web interface, REST API, command line • One-click “Globus Connect” install • 5-minute Globus Connect Multi User install
- 17. computationinstitute.org Early adoption isencouraging
- 18. computationinstitute.org Early adoption isencouraging 10,000 registered users; >100 daily ~18 PB moved; ~1B files 10x (or better) performance vs. scp 99.9% availability Entirely hosted on Amazon
- 19. 1e-011e+011e+031e+051e+07 duration 2011 2012 1 second 1minute 1 hour 1 day 1 week Duration of runs, in seconds, over time. Red: >10 TB transfer; green: >1 TB transfer.
- 20. computationinstitute.org K. Heitmann (Argonne) moves22 TB of cosmology data LANL ANL at 5 Gb/s
- 21. computationinstitute.org B. Winjum (UCLA)moves 900K-file plasma physics datasets UCLA NERSC
- 22. computationinstitute.org Dan Kozak (Caltech) replicates1 PB LIGO astronomy data for resilience
- 23. computationinstitute.org 2Credit: Kerstin Kleese-vanDam Erin Miller (PNNL) collects data at Advanced Photon Source, renders at PNNL, and views at ANL
- 24. computationinstitute.org • Collect • Move •Sync • Share • Analyze • Annotate • Publish • Search • Backup • Archive BIG DATA
- 25. computationinstitute.org • Collect • Move •Sync • Share • Analyze • Annotate • Publish • Search • Backup • Archive BIG DATA
- 26. computationinstitute.org Globus Online alreadydoes a lot Globus Toolkit Sharing Service Transfer Service Globus Nexus (Identity, Group, Profile) GlobusOnlineAPIs GlobusConnect
- 27. computationinstitute.org A platform forintegration
- 28. computationinstitute.org A platform forintegration
- 29. computationinstitute.org A platform forintegration
- 30. computationinstitute.org Data management SaaS(Globus) + Next-gen sequence analysis pipelines (Galaxy) + Cloud IaaS (Amazon) = Flexible, scalable, easy-to-use genomics analysis for all biologists globus genomics
- 31. computationinstitute.org31 Ravi Madduri, BoLiu, Paul Davé, et al.
- 32. computationinstitute.org RNA-Seq pipeline 3 Ravi Madduri,Bo Liu, Paul Davé, et al.
- 33. computationinstitute.org Amazon pricing forDiffusion Tensor Imaging pipeline 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 m1.large m1.xlarge m3.xlarge m3.2xlarge m2.xlarge m2.2xlarge m2.4xlarge CostperSubject($) On-Demand Spot (Low) Spot (High) Credit: Kyle Chard
- 34. computationinstitute.org We are addingcapabilities Globus Toolkit Sharing Service Transfer Service Globus Nexus (Identity, Group, Profile) GlobusOnlineAPIs GlobusConnect
- 35. computationinstitute.org Globus Toolkit Sharing Service TransferService Dataset Services Globus Nexus (Identity, Group, Profile) GlobusOnlineAPIs GlobusConnect We are adding capabilities
- 36. computationinstitute.org • Ingest andpublication – Imagine a DropBox that not only replicates, but also extracts metadata, catalogs, converts • Cataloging – Virtual views of data based on user-defined and/or automatically extracted metadata • Computation – Associate computational procedures, orchestrate application, catalog results, record provenance We are adding capabilities
- 37. computationinstitute.org Looking deeply athow researchers use data • A single research question often requires the integration of many data elements, that are: – In different locations – In different formats (Excel, text, CDF, HDF, …) – Described in different ways • Best grouping can vary during investigation – Longitudinal, vertical, cross-cutting • But always needs to be operated on as a unit – Share, annotate, process, copy, archive, …
- 38. computationinstitute.org How do wemanage data today? • Often, a curious mix of ad hoc methods – Organize in directories using file and directory naming conventions – Capture status in README files, spreadsheets, notebooks • Time-consuming, complex, error prone Why can’t we manage our data like we manage our pictures and music?
- 40. computationinstitute.org Introducing the dataset •Group data based on use, not location – Logical grouping to organize, reorganize, search, and describe usage • Tag with characteristics that reflect content … – Capture as much existing information as we can • …or to reflect current status in investigation – Stage of processing, provenance, validation, .. • Share data sets for collaboration – Control access to data and metadata • Operate on datasets as units – Copy, export, analyze, tag, archive, …
- 41. computationinstitute.org Builds on catalogas a service Approach • Hosted user-defined catalogs • Based on tag model <subject, name, value> • Optional schema constraints • Integrated with other Globus services Three REST APIs /query/ • Retrieve subjects /tags/ • Create, delete, retrieve tags /tagdef/ • Create, delete, retrieve tag definitions Builds on USC Tagfiler project (C. Kesselman et al.)
- 42. computationinstitute.org Provide more capabilityfor more people at lower cost by building a “Discovery Cloud” Delivering “Science as a service” Our vision for a 21st century discovery infrastructure
- 43. computationinstitute.org It’s a timeof great opportunity … to develop and apply Science aaS Globus Nexus (Identity, Group, Profile) … Sharing Service Transfer Service Dataset Services Globus Toolkit GlobusOnlineAPIs GlobusConnect
- 44. computationinstitute.org Thanks to greatcolleagues and collaborators • Steve Tuecke, Rachana Ananthakrishnan, Kyle Chard, Raj Kettimuthu, Ravi Madduri, Tanu Malik, and many others at Argonne & Uchicago • Carl Kesselman, Karl Czajkowski, Rob Schuler, and others at USC/ISI • Francesco de Carlo, Chris Jacobsen, and others at Argonne
- 45. computationinstitute.org Thank you toour sponsors!