Rethinking how we provide science ITin an era of    massive data but    modest budgetsIan Foster                          ...
Exploding data volumes in biology                        x107 in 14 years                                       www.ci.anl...
Exploding data volumes in astronomy      MACHO et al.: 1 TB     Palomar: 3 TB    2MASS: 10 TB    GALEX: 30 TB           10...
Exploding data volumes in climate science                     2004: 36 TB                     2012: 2,300 TBClimatemodel i...
The challenge of staying competitive"Well, in our country," said Alice … "youd generally get to somewhere else — if you ru...
Ways of running faster (1)                        Civilization advances by                        extending the number of ...
Ways of running faster (2)                 Utility computing                 “[t]he computing utility could become the bas...
Ways of running faster (3)                Collaboratories, P2P, crowdsourcing                Virtual organizations  Outsou...
Big science has been keeping up                                  OSG: 1.4M CPU-hours/day,                                 ...
But small science is strugglingMore data, more complex dataAd-hoc solutionsInadequate software, hardwareData plan mandates...
Medium science struggles too•        Dark Energy Survey            Blanco 4m on Cerro Tololo         receives 100,000 file...
Science IT crisis demands new approaches•    We have exceptional infrastructure for the 1%     (e.g., supercomputers, LHC,...
You can run a company from a coffee shop                                     www.ci.anl.gov13                             ...
Because businesses outsource their IT     Web presence     Email (hosted Exchange)     Calendar                       Soft...
And often their large-scale computing too     Web presence     Email (hosted Exchange)     Calendar                       ...
Consumers also outsource much of their IT
Let’s rethink how we provide research ITAccelerate discovery and innovation worldwideby providing research IT as a service...
Also address administrative costs?42% of the time spent by an average PIon a federally funded research project wasreported...
Time-consuming tasks in science•    Run experiments         • Communicate with•    Collect data              colleagues•  ...
Time-consuming tasks in science•    Run experiments         • Communicate with•    Collect data              colleagues•  ...
Scientific data delivery, 2012 1980•    “*A+ majority of users at BES facilities … physically transport data     to a home...
The challenge: Moving big data easilyWhat should be trivial …        “I need my data over there      Data                 ...
• GO PICTURE
GO-Transfer: Data transfer as SaaS• Reliable file transfer.      –   Easy “fire-and-forget” transfers      –   Automatic f...
Statistics and user feedback•        Launched November 2010          “Last time I needed to fetch                         ...
Common research data management steps     •   Dark Energy Survey   •   SBGrid structural biology consortium     •   Galaxy...
Towards “research IT as a service”           Scientific data management as a service     GO-Store      GO-Collaborate      ...
Research data management as a service•    GO-User           Today          •   GO-Store       Prototype     – Credentials ...
Collaboration Management                          www.ci.anl.gov                     30   www.ci.uchicago.edu
Other innovative science SaaS projects                                         www.ci.anl.gov32                           ...
Other innovative science SaaS projects                                         www.ci.anl.gov33                           ...
Other innovative science SaaS projects                                         www.ci.anl.gov34                           ...
Other innovative science SaaS projects                                         www.ci.anl.gov35                           ...
SaaS economics: A quick tutorial•    Lower per-user cost (x10 $     or more?) via aggregation     onto common     infrastr...
A 21st C science IT infrastructure strategy                                Small and medium laboratories and projects•    ...
Acknowledgments•    Colleagues at UChicago and Argonne     –   Steve Tuecke, Ravi Madduri, Kyle Chard, Tanu Malik,        ...
For more information•    www.globusonline.org; Twitter: @globusonline•    Foster, I. Globus Online: Accelerating and     d...
Thank you!foster@uchicago.eduwww.globusonline.orgTwitter: @globusonline, @ianfoster                                     ww...
Upcoming SlideShare
Loading in...5
×

Rethinking how we provide science IT in an era of massive data but modest budgets

2,709

Published on

A talk given in January 2012 at a wonderful conference organized in Zakopane, Poland, by colleagues from the erstwhile GridLab project. I talked about how increasing data volumes demand radically new approaches to delivering research computing. Lively discussion ensued.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,709
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
32
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • As in other outsourcing: benefits from specialization, economies of scale, reduced cost of meeting peak demand, flexibilityLivny: “I’ve been doing cloud computing since before it was called grid computing”
  • A particular strength of Grid has been in recognizing the need for infrastructure to support collaborative teaming
  • The concepts workThe technology worksBut groups still end up assembling verfically integrated solutions
  • PI and a handful of students and staff
  • The answer cannot simply be more moneyWe lack both $$ and the people to spend $$ on
  • Key points: intuitive interfaces, no local software, positive returns to scaleWe live in a strange time technologically. In our homes, we have enormously sophisticated digital media management technology. Intuitive, automated, high-performance discovery and streaming—Netflix and iTunes, for example.
  • Not (particularly) computing as a serviceBut the IT functions that researchers need to functionInclude collaboration as a service
  • Note that large-scale computing is an important part of the picture for manyBut the MOST important issues are often more mundane—keeping track of data, sharing data with others, finding relevant software, …
  • But when we get to work, we go back in time 20 years
  • Transcript of "Rethinking how we provide science IT in an era of massive data but modest budgets"

    1. 1. Rethinking how we provide science ITin an era of massive data but modest budgetsIan Foster www.ci.anl.gov www.ci.uchicago.edu
    2. 2. Exploding data volumes in biology x107 in 14 years www.ci.anl.gov2 www.ci.uchicago.edu
    3. 3. Exploding data volumes in astronomy MACHO et al.: 1 TB Palomar: 3 TB 2MASS: 10 TB GALEX: 30 TB 100,000 TB Sloan: 40 TBPan-STARRS: 40,000 TB www.ci.anl.gov3 www.ci.uchicago.edu
    4. 4. Exploding data volumes in climate science 2004: 36 TB 2012: 2,300 TBClimatemodel intercomparisonproject (CMIP) of the IPCC www.ci.anl.gov4 www.ci.uchicago.edu
    5. 5. The challenge of staying competitive"Well, in our country," said Alice … "youd generally get to somewhere else — if you run very fast for a long time, as weve been doing.”"A slow sort of country!" said the Queen. "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!" www.ci.anl.gov5 www.ci.uchicago.edu
    6. 6. Ways of running faster (1) Civilization advances by extending the number of important operations which we can perform without thinking about them Alfred North Whitehead, 1911 Enhance human capabilities www.ci.anl.gov6 www.ci.uchicago.edu
    7. 7. Ways of running faster (2) Utility computing “[t]he computing utility could become the basis for a new and important industry” – McCarthy, 1960 Outsourceautomatable Grid computing tasks “provide access to computing on demand” – The Grid: Blueprint for a New Computing Inf., 1999 Cloud computing “delivery of computing as a service rather than a product” *Wikipedia, 2012+ Enhance human capabilities www.ci.anl.gov 7 www.ci.uchicago.edu
    8. 8. Ways of running faster (3) Collaboratories, P2P, crowdsourcing Virtual organizations Outsource “flexible, secure, coordinated resource sharingautomatable among dynamic collections of individuals, tasks institutions, and resources”, Anatomy of Grid, 2001 Join forces with others Enhance human capabilities www.ci.anl.gov 8 www.ci.uchicago.edu
    9. 9. Big science has been keeping up OSG: 1.4M CPU-hours/day, >90 sites, >3000 users, >260 pubs in 2010LIGO: 1 PB data in last sciencerun, distributed worldwide Robust production solutions Substantial teams and expense Sustained, multi-year effort Application-specific solutions, built on common technology ESG: 1.2 PB climate data delivered to 23,000 users; 600+ pubs www.ci.anl.gov 9 www.ci.uchicago.edu
    10. 10. But small science is strugglingMore data, more complex dataAd-hoc solutionsInadequate software, hardwareData plan mandates www.ci.anl.gov10 www.ci.uchicago.edu
    11. 11. Medium science struggles too• Dark Energy Survey Blanco 4m on Cerro Tololo receives 100,000 files each night in Illinois• They transmit files to Texas for analysis … then move results back to Illinois• Process must be reliable, routine, and efficient• The IT team is not large Image credit: Roger Smith/NOAO/AURA/NSF www.ci.anl.gov 11 www.ci.uchicago.edu
    12. 12. Science IT crisis demands new approaches• We have exceptional infrastructure for the 1% (e.g., supercomputers, LHC, …)• But not for the 99% (e.g., the vast majority of the 1.8M publicly funded researchers in the EU) We need new approaches to providing science IT, that: — Reduce barriers to entry — Are cheaper — Are sustainable www.ci.anl.gov12 www.ci.uchicago.edu
    13. 13. You can run a company from a coffee shop www.ci.anl.gov13 www.ci.uchicago.edu
    14. 14. Because businesses outsource their IT Web presence Email (hosted Exchange) Calendar Software Telephony (hosted VOIP) as a Service Human resources and payroll (SaaS) Accounting Customer relationship mgmt www.ci.anl.gov14 www.ci.uchicago.edu
    15. 15. And often their large-scale computing too Web presence Email (hosted Exchange) Calendar Software Telephony (hosted VOIP) as a Service Human resources and payroll (SaaS) Accounting Customer relationship mgmt Infrastructure Data analytics as a Service Content distribution (IaaS) www.ci.anl.gov15 www.ci.uchicago.edu
    16. 16. Consumers also outsource much of their IT
    17. 17. Let’s rethink how we provide research ITAccelerate discovery and innovation worldwideby providing research IT as a serviceLeverage software-as-a-service to• provide millions of researchers with unprecedented access to powerful tools;• enable a massive shortening of cycle times in time-consuming research processes; and• reduce research IT costs dramatically via economies of scale—and address sustainability? www.ci.anl.gov17 www.ci.uchicago.edu
    18. 18. Also address administrative costs?42% of the time spent by an average PIon a federally funded research project wasreported to be expended on administrativetasks related to that project rather than onresearch — Federal Demonstration Partnership faculty burden survey, 2007 www.ci.anl.gov18 www.ci.uchicago.edu
    19. 19. Time-consuming tasks in science• Run experiments • Communicate with• Collect data colleagues• Manage data • Publish papers• Move data • Find, configure, install• Acquire computers relevant software• Analyze data • Find, access, analyze relevant data• Run simulations • Order supplies• Compare experiment with simulation • Write proposals• Search the literature • Write reports • … www.ci.anl.gov19 www.ci.uchicago.edu
    20. 20. Time-consuming tasks in science• Run experiments • Communicate with• Collect data colleagues• Manage data • Publish papers• Move data • Find, configure, install• Acquire computers relevant software• Analyze data • Find, access, analyze relevant data• Run simulations • Order supplies• Compare experiment with simulation • Write proposals• Search the literature • Write reports • … www.ci.anl.gov20 www.ci.uchicago.edu
    21. 21. Scientific data delivery, 2012 1980• “*A+ majority of users at BES facilities … physically transport data to a home institution using portable media … data volumes are going to increase significantly in the next few years (to 70 TB/day or more) – data must be transferred over the network”• “the effectiveness of data transfer middleware [is] not just on the transfer speed, but also the time and interruption to other work required to supervise and check on the success of large data transfers”• “It took two weeks and email traffic between network specialists at NERSC and ORNL, sys-admins at NERSC, … and combustion staff at ORNL and SNL to move 10 TB from NERSC to ORNL” Major usability, productivity, performance problems [ESNet Network Requirements Workshops, 2007-2010] www.ci.anl.gov21 www.ci.uchicago.edu
    22. 22. The challenge: Moving big data easilyWhat should be trivial … “I need my data over there Data Data – at my _____” ( Source Destination supercomputing center, campus server, etc.) … can be painfully tedious and time-consuming “GAAAH !%&@#& ” ! Config issues Data Data ! Firewall issues Source Destination ! Unexpected failure = manual retry www.ci.anl.gov22 www.ci.uchicago.edu
    23. 23. • GO PICTURE
    24. 24. GO-Transfer: Data transfer as SaaS• Reliable file transfer. – Easy “fire-and-forget” transfers – Automatic fault recovery – High performance – Across multiple security domains• No IT required. – Software as a Service (SaaS) • No client software installation • New features automatically available – Consolidated support & troubleshooting – Works with existing GridFTP servers – Globus Connect solves “last mile problem”GO-Transfer is the initial offering of the US NationalScience Foundation’s XSEDE User Access Services (XUAS) www.ci.anl.gov 24 www.ci.uchicago.edu
    25. 25. Statistics and user feedback• Launched November 2010 “Last time I needed to fetch 100,000 files from NERSC, a >3500 users registered graduate student babysat the >2500 TB user data moved process for a month.” >130 million user files moved “I expected to spend four >300 endpoints registered weeks writing code to manage my data transfers; with Globus• Widely used on TeraGrid/ Online, I was up and running in five minutes.” XSEDE; other centers & facilities; internationally “Transferred my data in 20 minutes instead of 61 hours.• >20x faster than SCP Makes these global climate• Comparable to hand-tuned simulations manageable.” www.ci.anl.gov 26 www.ci.uchicago.edu
    26. 26. Common research data management steps • Dark Energy Survey • SBGrid structural biology consortium • Galaxy genomics • NCAR climate data applications • LIGO observatory • Land use change; economics www.ci.anl.gov27 www.ci.uchicago.edu
    27. 27. Towards “research IT as a service” Scientific data management as a service GO-Store GO-Collaborate GO-Galaxy GO-Transfer GO-Compute GO-Catalog GO-Team GO-User www.ci.anl.gov28 www.ci.uchicago.edu
    28. 28. Research data management as a service• GO-User Today • GO-Store Prototype – Credentials and other – Access to campus, profile information cloud, XSEDE storage• GO-Transfer • GO-Catalog – On-demand metadata – Data movement catalogs• GO-Team Beta • GO-Compute – Group membership – Access to computers• GO-Collaborate • GO-Galaxy – Connect to collaborative – Share, create, run tools: Jira, Confluence, … workflows www.ci.anl.gov29 www.ci.uchicago.edu
    29. 29. Collaboration Management www.ci.anl.gov 30 www.ci.uchicago.edu
    30. 30. Other innovative science SaaS projects www.ci.anl.gov32 www.ci.uchicago.edu
    31. 31. Other innovative science SaaS projects www.ci.anl.gov33 www.ci.uchicago.edu
    32. 32. Other innovative science SaaS projects www.ci.anl.gov34 www.ci.uchicago.edu
    33. 33. Other innovative science SaaS projects www.ci.anl.gov35 www.ci.uchicago.edu
    34. 34. SaaS economics: A quick tutorial• Lower per-user cost (x10 $ or more?) via aggregation onto common infrastructure• Initial “cost trough” due 0 to fixed costs Time• Per-user revenue permits positive return to scale Lower per-user costs• Further reduce per-user suggest new approaches cost over time to sustainability www.ci.anl.gov36 www.ci.uchicago.edu
    35. 35. A 21st C science IT infrastructure strategy Small and medium laboratories and projects• To provide L L L L L L L L L more capability for L L P L PL L P L P L L P L L L L L L L L L L more people at less cost …• Create infrastructure Research data management a – Robust and universal Collaboration, computation a – Economies of scale Research administration S – Positive returns to scale• Via the creative use of – Aggregation (“cloud”) – Federation (“grid”) www.ci.anl.gov 37 www.ci.uchicago.edu
    36. 36. Acknowledgments• Colleagues at UChicago and Argonne – Steve Tuecke, Ravi Madduri, Kyle Chard, Tanu Malik, Rachana Ananthakrisnan, Raj Kettimuthu, and others listed at www.globusonline.org/about/goteam/• Carl Kesselman and other colleagues at other institutions• Participants in the recent ICiS workshop on “Human-Computer Symbiosis: 50 Years On”• NSF OCI and MPS; DOE ASCR; NIH for support www.ci.anl.gov38 www.ci.uchicago.edu
    37. 37. For more information• www.globusonline.org; Twitter: @globusonline• Foster, I. Globus Online: Accelerating and democratizing science through cloud-based services. IEEE Internet Computing(May/June):70-73, 2011.• Allen, B., Bresnahan, J., Childers, L., Foster, I., Kandaswamy, G., Kettimuthu, R., Kordas, J., Link, M., Martin, S., Pickett, K. and Tuecke, S. Software as a Service for Data Scientists. Communications of the ACM, Feb, 2012. www.ci.anl.gov39 www.ci.uchicago.edu
    38. 38. Thank you!foster@uchicago.eduwww.globusonline.orgTwitter: @globusonline, @ianfoster www.ci.anl.gov www.ci.uchicago.edu
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×