Rethinking how we provide science IT in an era of massive data but modest budgets

Rethinking how we provide science ITin an era of massive data but modest budgetsIan Foster www.ci.anl.gov www.ci.uchicago.edu

Exploding data volumes in biology x107 in 14 years www.ci.anl.gov2 www.ci.uchicago.edu

Exploding data volumes in astronomy MACHO et al.: 1 TB Palomar: 3 TB 2MASS: 10 TB GALEX: 30 TB 100,000 TB Sloan: 40 TBPan-STARRS: 40,000 TB www.ci.anl.gov3 www.ci.uchicago.edu

Exploding data volumes in climate science 2004: 36 TB 2012: 2,300 TBClimatemodel intercomparisonproject (CMIP) of the IPCC www.ci.anl.gov4 www.ci.uchicago.edu

The challenge of staying competitive"Well, in our country," said Alice … "youd generally get to somewhere else — if you run very fast for a long time, as weve been doing.”"A slow sort of country!" said the Queen. "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!" www.ci.anl.gov5 www.ci.uchicago.edu

Ways of running faster (1) Civilization advances by extending the number of important operations which we can perform without thinking about them Alfred North Whitehead, 1911 Enhance human capabilities www.ci.anl.gov6 www.ci.uchicago.edu

Ways of running faster (2) Utility computing “[t]he computing utility could become the basis for a new and important industry” – McCarthy, 1960 Outsourceautomatable Grid computing tasks “provide access to computing on demand” – The Grid: Blueprint for a New Computing Inf., 1999 Cloud computing “delivery of computing as a service rather than a product” *Wikipedia, 2012+ Enhance human capabilities www.ci.anl.gov 7 www.ci.uchicago.edu

Ways of running faster (3) Collaboratories, P2P, crowdsourcing Virtual organizations Outsource “flexible, secure, coordinated resource sharingautomatable among dynamic collections of individuals, tasks institutions, and resources”, Anatomy of Grid, 2001 Join forces with others Enhance human capabilities www.ci.anl.gov 8 www.ci.uchicago.edu

Big science has been keeping up OSG: 1.4M CPU-hours/day, >90 sites, >3000 users, >260 pubs in 2010LIGO: 1 PB data in last sciencerun, distributed worldwide Robust production solutions Substantial teams and expense Sustained, multi-year effort Application-specific solutions, built on common technology ESG: 1.2 PB climate data delivered to 23,000 users; 600+ pubs www.ci.anl.gov 9 www.ci.uchicago.edu

But small science is strugglingMore data, more complex dataAd-hoc solutionsInadequate software, hardwareData plan mandates www.ci.anl.gov10 www.ci.uchicago.edu

Medium science struggles too• Dark Energy Survey Blanco 4m on Cerro Tololo receives 100,000 files each night in Illinois• They transmit files to Texas for analysis … then move results back to Illinois• Process must be reliable, routine, and efficient• The IT team is not large Image credit: Roger Smith/NOAO/AURA/NSF www.ci.anl.gov 11 www.ci.uchicago.edu

Science IT crisis demands new approaches• We have exceptional infrastructure for the 1% (e.g., supercomputers, LHC, …)• But not for the 99% (e.g., the vast majority of the 1.8M publicly funded researchers in the EU) We need new approaches to providing science IT, that: — Reduce barriers to entry — Are cheaper — Are sustainable www.ci.anl.gov12 www.ci.uchicago.edu

You can run a company from a coffee shop www.ci.anl.gov13 www.ci.uchicago.edu

Because businesses outsource their IT Web presence Email (hosted Exchange) Calendar Software Telephony (hosted VOIP) as a Service Human resources and payroll (SaaS) Accounting Customer relationship mgmt www.ci.anl.gov14 www.ci.uchicago.edu

And often their large-scale computing too Web presence Email (hosted Exchange) Calendar Software Telephony (hosted VOIP) as a Service Human resources and payroll (SaaS) Accounting Customer relationship mgmt Infrastructure Data analytics as a Service Content distribution (IaaS) www.ci.anl.gov15 www.ci.uchicago.edu

Consumers also outsource much of their IT

Let’s rethink how we provide research ITAccelerate discovery and innovation worldwideby providing research IT as a serviceLeverage software-as-a-service to• provide millions of researchers with unprecedented access to powerful tools;• enable a massive shortening of cycle times in time-consuming research processes; and• reduce research IT costs dramatically via economies of scale—and address sustainability? www.ci.anl.gov17 www.ci.uchicago.edu

Also address administrative costs?42% of the time spent by an average PIon a federally funded research project wasreported to be expended on administrativetasks related to that project rather than onresearch — Federal Demonstration Partnership faculty burden survey, 2007 www.ci.anl.gov18 www.ci.uchicago.edu

Time-consuming tasks in science• Run experiments • Communicate with• Collect data colleagues• Manage data • Publish papers• Move data • Find, configure, install• Acquire computers relevant software• Analyze data • Find, access, analyze relevant data• Run simulations • Order supplies• Compare experiment with simulation • Write proposals• Search the literature • Write reports • … www.ci.anl.gov19 www.ci.uchicago.edu

Time-consuming tasks in science• Run experiments • Communicate with• Collect data colleagues• Manage data • Publish papers• Move data • Find, configure, install• Acquire computers relevant software• Analyze data • Find, access, analyze relevant data• Run simulations • Order supplies• Compare experiment with simulation • Write proposals• Search the literature • Write reports • … www.ci.anl.gov20 www.ci.uchicago.edu

Scientific data delivery, 2012 1980• “*A+ majority of users at BES facilities … physically transport data to a home institution using portable media … data volumes are going to increase significantly in the next few years (to 70 TB/day or more) – data must be transferred over the network”• “the effectiveness of data transfer middleware [is] not just on the transfer speed, but also the time and interruption to other work required to supervise and check on the success of large data transfers”• “It took two weeks and email traffic between network specialists at NERSC and ORNL, sys-admins at NERSC, … and combustion staff at ORNL and SNL to move 10 TB from NERSC to ORNL” Major usability, productivity, performance problems [ESNet Network Requirements Workshops, 2007-2010] www.ci.anl.gov21 www.ci.uchicago.edu

The challenge: Moving big data easilyWhat should be trivial … “I need my data over there Data Data – at my _____” ( Source Destination supercomputing center, campus server, etc.) … can be painfully tedious and time-consuming “GAAAH !%&@#& ” ! Config issues Data Data ! Firewall issues Source Destination ! Unexpected failure = manual retry www.ci.anl.gov22 www.ci.uchicago.edu

• GO PICTURE

GO-Transfer: Data transfer as SaaS• Reliable file transfer. – Easy “fire-and-forget” transfers – Automatic fault recovery – High performance – Across multiple security domains• No IT required. – Software as a Service (SaaS) • No client software installation • New features automatically available – Consolidated support & troubleshooting – Works with existing GridFTP servers – Globus Connect solves “last mile problem”GO-Transfer is the initial offering of the US NationalScience Foundation’s XSEDE User Access Services (XUAS) www.ci.anl.gov 24 www.ci.uchicago.edu

Statistics and user feedback• Launched November 2010 “Last time I needed to fetch 100,000 files from NERSC, a >3500 users registered graduate student babysat the >2500 TB user data moved process for a month.” >130 million user files moved “I expected to spend four >300 endpoints registered weeks writing code to manage my data transfers; with Globus• Widely used on TeraGrid/ Online, I was up and running in five minutes.” XSEDE; other centers & facilities; internationally “Transferred my data in 20 minutes instead of 61 hours.• >20x faster than SCP Makes these global climate• Comparable to hand-tuned simulations manageable.” www.ci.anl.gov 26 www.ci.uchicago.edu

Common research data management steps • Dark Energy Survey • SBGrid structural biology consortium • Galaxy genomics • NCAR climate data applications • LIGO observatory • Land use change; economics www.ci.anl.gov27 www.ci.uchicago.edu

Towards “research IT as a service” Scientiﬁc data management as a service GO-Store GO-Collaborate GO-Galaxy GO-Transfer GO-Compute GO-Catalog GO-Team GO-User www.ci.anl.gov28 www.ci.uchicago.edu

Research data management as a service• GO-User Today • GO-Store Prototype – Credentials and other – Access to campus, profile information cloud, XSEDE storage• GO-Transfer • GO-Catalog – On-demand metadata – Data movement catalogs• GO-Team Beta • GO-Compute – Group membership – Access to computers• GO-Collaborate • GO-Galaxy – Connect to collaborative – Share, create, run tools: Jira, Confluence, … workflows www.ci.anl.gov29 www.ci.uchicago.edu

Collaboration Management www.ci.anl.gov 30 www.ci.uchicago.edu

Other innovative science SaaS projects www.ci.anl.gov32 www.ci.uchicago.edu

SaaS economics: A quick tutorial• Lower per-user cost (x10 $ or more?) via aggregation onto common infrastructure• Initial “cost trough” due 0 to fixed costs Time• Per-user revenue permits positive return to scale Lower per-user costs• Further reduce per-user suggest new approaches cost over time to sustainability www.ci.anl.gov36 www.ci.uchicago.edu

A 21st C science IT infrastructure strategy Small and medium laboratories and projects• To provide L L L L L L L L L more capability for L L P L PL L P L P L L P L L L L L L L L L L more people at less cost …• Create infrastructure Research data management a – Robust and universal Collaboration, computation a – Economies of scale Research administration S – Positive returns to scale• Via the creative use of – Aggregation (“cloud”) – Federation (“grid”) www.ci.anl.gov 37 www.ci.uchicago.edu

Acknowledgments• Colleagues at UChicago and Argonne – Steve Tuecke, Ravi Madduri, Kyle Chard, Tanu Malik, Rachana Ananthakrisnan, Raj Kettimuthu, and others listed at www.globusonline.org/about/goteam/• Carl Kesselman and other colleagues at other institutions• Participants in the recent ICiS workshop on “Human-Computer Symbiosis: 50 Years On”• NSF OCI and MPS; DOE ASCR; NIH for support www.ci.anl.gov38 www.ci.uchicago.edu

For more information• www.globusonline.org; Twitter: @globusonline• Foster, I. Globus Online: Accelerating and democratizing science through cloud-based services. IEEE Internet Computing(May/June):70-73, 2011.• Allen, B., Bresnahan, J., Childers, L., Foster, I., Kandaswamy, G., Kettimuthu, R., Kordas, J., Link, M., Martin, S., Pickett, K. and Tuecke, S. Software as a Service for Data Scientists. Communications of the ACM, Feb, 2012. www.ci.anl.gov39 www.ci.uchicago.edu

Thank you!foster@uchicago.eduwww.globusonline.orgTwitter: @globusonline, @ianfoster www.ci.anl.gov www.ci.uchicago.edu

Rethinking how we provide science IT in an era of massive data but modest budgets

by Ian Foster on Feb 23, 2012

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 2

Statistics

Rethinking how we provide science IT in an era of massive data but modest budgets Presentation Transcript

http://us-w1.rockmelt.com	1
http://www.onlydoo.com	1