ORION Workshop: XSEDE and Building a National/International Cyberinfrastructure


Published on

Title: XSEDE and Building a National/International Cyberinfrastructure

In this talk I will present a high level overview of XSEDE from the point of view of building a national/international scale cyberinfrastructure and the associated collaborations necessary. If nothing else, XSEDE has been an adventure in social engineering and I will comment on some of the aspects of these highly distributed and complex interactions. Along the way I will share some of the pitfalls, lessons learned and continuing challenges.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

ORION Workshop: XSEDE and Building a National/International Cyberinfrastructure

  1. 1. March 20, 2014 XSEDE and Building a National/International Cyberinfrastructure John Towns PI and Project Director, XSEDE Director, Collaborative Cyberinfrastructure Programs, NCSA jtowns@ncsa.illinois.edu
  2. 2. License terms • Please cite as: Towns, John. 2014 XSEDE and Building a National/International Cyberinfrastructure, Presentation, Advanced Computing – Transforming Research: A Planning Workshop, 13 Feb 2014, Toronto, Ontario, Canada, http://www.slideshare.net/jtownsil/xsede-at-orionworkshopfeb2014. • ORCID ID: http://orcid.org/0000-0001-7961-2277 • Except where otherwise noted, by inclusion of a source URL or some other note, the contents of this presentation are © by the Board of Trustees of University of Illinois. This content is released under the Creative Commons Attribution 3.0 Unported license (http://creativecommons.org/licenses/by/3.0/). This license includes the following terms: You are free to share – copy and redistribute the material in any medium or format; and to adapt – remix, transform, and build upon the material for any purpose, even commercially. • This can be done under the following conditions: attribution – You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
  3. 3. Who is this guy? • John Towns – PI and Project Director of XSEDE – Director, Collaborative eScience Program Office at NCSA – Co-Founder, Illinois Campus Cluster Program • Background – just a mid-western boy from Missouri • I don’t see a lot of folks from Missouri in the circles I am in – failed physicist • general relativity – failed computational scientist • numerical simulation of black hole spacetimes – failed networked applications guy • NLANR-DAST: National Laboratory for Applied Network Research- Distributed Applications Support Team – but, a budding social engineer • Wearing my XSEDE hat for this discussion 3
  4. 4. Who is this guy, part deux? full disclosure/caveats/disclaimers • To be clear – I am a member of the Compute Canada Board of Directors – I am not here in that capacity • I am not in a position to speak for the CC Board – I may make comments regarding the CC Board if appropriate – I am open to comments from the community that I can carry back to the CC Board • My capacity here – PI and Project director of XSEDE – someone with a lot of experience in building institutional/regional/national/international cyberinfrastructure 4
  5. 5. Step-by-Step-Instructions • How to create an open cyberinfrastructure ecosystem in 1,200 easy steps 5 . . . .
  6. 6. Boundary Conditions and Principles for XSEDE • XSEDE inherited TeraGrid environment • XSEDE inherited TG community and their expectations • Point of view has changed – not an HPC/CS/tech play – about productivity and creating the environment necessary to be productive • focus on the success of researchers! • Finally figured out that the project must define a solution that is designed to evolve! – technologically and organizationally! • Identify the greatest needs and start there – Don’t forget what you have learned – both good and bad! • Oh yea… the researchers don’t care about your existence (per se) – they care about access to resources, services and support 6
  7. 7. XSEDE – accelerating scientific discovery • XSEDE’s Vision: a world of digitally enabled researchers, engineers, and scholars participating in multidisciplinary collaborations to tackle society’s grand challenges • XSEDE’s Mission: to substantially enhance the productivity of a growing community of researchers, engineers, and scholars through access to advanced digital services that support open research 7
  8. 8. XSEDE’s Strategic Goals • Deepen and extend the use of the advanced digital research services ecosystem – deepen use by existing researchers, engineers, and scholars – extend use to new communities – prepare the current and next generation via education, training, and outreach – raise the general awareness of the value of advanced digital services • Advance the advanced digital research services ecosystem – create an open and evolving e-infrastructure – enhance the array of technical expertise and support services offered • Sustain the advanced digital research services ecosystem – assure and maintain a reliable and secure infrastructure – provide excellent user support services – operate an effective and innovative virtual organization 8
  9. 9. What is XSEDE? • An ecosystem of advanced digital services accelerating scientific discovery – support a growing portfolio of resources and services • advanced computing, high-end visualization, data analysis, and other resources and services • interoperability with other infrastructures • A virtual organization (partnership!) providing – dynamic distributed infrastructure – support services, and technical expertise to enable researchers engineers and scholars • addressing the most important and challenging problems facing the nation and world • A project funded by the National Science Foundation 9
  10. 10. XSEDE Factoids: high order bits • 5 year, US$121M project – plus US$9M, 5 year Technology Investigation Service • separate award from NSF – option for additional 5 years of funding upon major review after PY3 • No funding for major hardware – coordination, support and creating a national/international cyberinfrastructure – coordinate allocations, support, training and documentation for >$100M of concurrent project awards from NSF • ~140 FTE /~250 individuals funded across 20 partner institutions – this requires solid partnering! 10
  11. 11. Convenience requirements will always increase Each generation of users requires more convenience than the former: thus we must always be adding new layers of software while maintaining and extending existing reliability and capability. Change is the only Constant – Heraclitis 535BC-475BC 11 No, his mind is not for rent To any god or government. Always hopeful, yet discontent, He knows changes aren't permanent, But change is. – Rush - Tom Sawyer
  12. 12. Some Important Lessons Learned • TeraGrid Project Final Report – http://hdl.handle.net/2142/43874 – a “lessons learned” document from the TeraGrid experience • Some highlights that speak to partnership needs – “The work of managing such a large project needs to be carefully distributed to make sure all the work gets done, but also needs to be well coordinated, to make sure there is a consistent message.” – “TeraGrid lacked a structure to make contentious decisions and the authority to enforce those decisions even when some participants disagreed.” – “The heterogeneity of the user environments on the various computing systems was a difficult challenge for all stakeholders.” – “TeraGrid was able to sustain and scale-up projects through highly leveraged funding from other grants raised by the RPs, particularly in the education realm.” 12
  13. 13. Some Unexpected Challenges: XSEDE is a socio-technical ecosystem • Highly distributed organization – challenges in managing a project that involves staff at 20 partner institutions • A completely virtual organization – breaking new ground from an organizational structure and management point of view • Highly distributed engineering project – developing new methodologies to adapt traditional practices to the unusual context of XSEDE 13
  14. 14. Some Musings on Cyberinfrastructure Partnerships Work • Partnerships can be a Good Thing – leverage of resources, services are personnel – alignment of multiple efforts and funding steams to accomplish larger goals – ultimately facilitate greater productivity of researchers! • But there are practical pitfalls – socio-political issues often obscure the real goal – strong individual personalities can upset the applecart in a variety of ways – the importance of the mechanics of managing the partnership and maintaining accountability among the partners can never be underestimated 14
  15. 15. Strategies for Success in (Cyberinfrastructure) Partnerships • Keep your eye on the ball! – vision, mission, goals, KPIs, metrics – know why the partnership exists and how you will assess your collective success – all partners must support the vision • Know your stakeholders and their needs/interests/priorities – all partners are also stakeholders, not just the research community being supported • KISS principle – hierarchies of partnerships seriously complicate the management of the whole enterprise • coordination of efforts, reporting, accountability, flowing money, … – institutions must have the freedom to partner locally, regionally, nationally and internationally in order to pursue their objectives and priorities • Make sure you can make decisions! – decision-making processes that do not require 100% consensus but are also inclusive of the stakeholders are key 15
  16. 16. Questions?