ISC Cloud13 Sill - Crossing organizational boundaries in cloud computing


Published on

Talk given at ISC Cloud'13: HPC and Manufacturing Meet Cloud, held 23-24 Sep 2013 in Heidelberg, Germany.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

ISC Cloud13 Sill - Crossing organizational boundaries in cloud computing

  1. 1. The Conference on the Use of Cloud for High Performance Computing (HPC) Taking the Best from Industry & Academics: Cross-SDO Projects to Accelerate HPC Cloud Innovation Alan Sill, Texas Tech University; VP of Standards, Open Grid Forum
  2. 2. Alan Sill, TTU September 24, 2013 About the Open Grid Forum: Open Grid Forum (OGF) is a leading global standards organization cooperating with many partners in the areas of cloud, grid and related forms of advanced distributed computing. The OGF community and its partners pursue these topics through an open process for development, creation and promotion of relevant specifications and use cases. The central feature of this work is open forum with open processes to champion architectural blueprints related to cloud and grid computing. The resulting specifications and standards enable pervasive adoption of advanced distributed computing techniques for business and research worldwide. 2
  3. 3. Alan Sill, TTU September 24, 2013 History and Background • OGF began in 2001 as an organization to promote the advancement of distributed computing worldwide. • Grid Forum --> Global Grid Forum --> GGF + Enterprise Grid Alliance --> formation of OGF in 2005. • Mandate is to take on all forms of distributed computing and to work to promote cooperation, information exchange, best practices in use and standardization. • OGF best known for a series of important computing, security and network standards that form the basis for major science and business-based distributed computing (BES, GridFTP, DRMAA, JSDL, RNS, GLUE, UR, etc.). • Have also been working on cloud and Big Data standards (OCCI, WS-Agreement, DFDL, etc.) for several years. • Cooperative work agreements with other SDOs in place. 3
  4. 4. Alan Sill, TTU September 24, 2013 • DRMAA: Distributed Resource Management Application API Grid Engine, Open Grid Scheduler: (open source); TORQUE and related products: Adaptive Computing; PBS Works: Altair Engineering; Gridway: DSA Research; Condor: U. of Wisconsin / Red Hat; • OGSA® Basic Execution Service Version 1.0 and BES HPC Profile: BES++ for LSF/SGE/PBS: Platform Computing; Windows HPC Server 2008: Microsoft Corporation; PBS Works - (client only): Altair Engineering; • JSDL: Job Submission Description Language (family of specs): BES++ for LSF/SGE/PBS and Platform LSF: Platform Computing; Windows HPC Server 2008: Microsoft Corporation; PBS Works - (client only): Altair Engineering; • WS-Agreement (family of specifications): ElasticLM License-as-a-Service: ElasticLM; BEinGrid SLA Negotiator, LM-Architecture and Framework: (Multiple partners); BREIN SLA Management Framework: (Multiple partners); WSAG4J, Web Services Agreement for Java (framework implementation): Fraunhofer SCAI. 4 OGF Standards In Use In Industry:
  5. 5. Alan Sill, TTU September 24, 2013 OGF Standards OGF has an extensive set of applicable standards related to advanced distributed grid and cloud computing and associated storage management and network operation: • Job Submission and Workflow Management (JSDL, BES) • Federated Identity Management (FedSec-CG) • Virtual Organizations (VOMS) • Managing the Trust Eco-System (CA operations, AuthN/AuthZ) • Network Management (NSI, NML, NMC, NM) • Secure, fast multi--party data transfer (GridFTP, SRM) • Service Agreements (WS-Agreement, WS-Agreement Negotiation) • Data Format Description (DFDL) • Cloud Computing interfaces (OCCI) • Distributed resource management APIs (DRMAA, SAGA, etc.) • Firewall Traversal (FiTP) • (Many others under development) 5
  6. 6. Alan Sill, TTU September 24, 2013 Example: Worldwide LHC Computing Grid 6 ~450,000 cpu cores ~430 Pb storage Typical data transfer rate: ~12 GByte/sec Total worldwide grid capacity: ~2x WLCG across all grids and VOs
  7. 7. LSN-MAGIC Meeting February 22, 2012 The Role of Standards for Risk Reduction and Inter-operation in XSEDE Andrew Grimshaw XSEDE: The Next Generation of US Supercomputing Infrastructure OGF standards power some of the largest supercomputing infrastructures in the world! XSEDE cloud infrastructure still being written
  8. 8. LSN-MAGIC Meeting February 22, 2012 XSEDE Services Layer: Simple services combined in many ways 8 –Resource  Namespace  Service  1.1 –OGSA  Basic  Execu8on  Service –OGSA  WSRF  BP  –  metadata  and  no8fica8on –OGSA-­‐ByteIO –GridFTP –JSDL,  BES,  BES  HPC  Profile –WS  Trust  Secure  Token  Services –WSI  BSP  for  transport  of  creden8als –…  (more  than  we  have  room  to  cover  here) Examples – (not a complete list) Basic message: XSEDE represents a phase change in the engagement of OGF standards with US cyberinfrastructure.
  9. 9. Alan Sill, TTU September 24, 2013 9 • OCCI is an API and Protocol • Sits on the boundary of a Service Provider and Service Consumer • No assumptions about the boundary OCCI® by OGF Open Cloud Computing Interface® Many real-world implementations!
  10. 10. Task Force – Sept 2013 Members •70 individuals •40 institutions •13 countries Stakeholders •23 Resource Providers •10 Technology Providers •7 User Communities •4 Liaisons Technologies •OpenNebula. •StratusLab. •OpenStack. •Synnefo. •WNoDeS. BSC CNRS LMU OeRC Masaryk TUD IFAE Cyfronet 100%IT CESNET RADICAL SRCE DANTE FZJ GRNET GWDG STFC SARA KTH INFNFCTSG Imperial CESGACETA IFCA IGI IPHC IN2P3 SZTAKI IISAS SixSq Credit: David Wallom Chair EGI Federated Cloud Task Force
  11. 11. Federation Test bed – September 2013 FedCloud Core Services Information System Top-DBII Monitoring Nagios Image Metadata Marketplace Appliance Repository VO VOMS server User Interfaces OCCI Clients rOCCI;WNoDeS-CLI CDMI Clients Libcdmi-java Resource Providers VOMS proxy Vmcatcher Vmcaster Marketplace client SSM CDMI server LDAP OpenNebula OpenStack StratusLab WNoDeS OCCI server AAI VOMS Proxy EGI Production Infrastructure Service Availability SAM Accounting APEL Configuration DB GOCDB External Services Certification Authorities Credit: David Wallom Chair EGI Federated Cloud Task Force
  12. 12. Alan Sill, TTU September 24, 2013 12 Collaborating Organizations: • It is often said that there are “too many standards organizations”. This is a lot like saying there is “too much software”. • Each has its own area of specialty, its own contributor base, and its own method of funding to develop its work products. • How does OGF relate to these organizations? (Answer: Cooperatively!) (Some of many...)
  13. 13. Interoperability: Appropriate use of standards permits software processes to interoperate with other components and infrastructures, and thus reduces risks including the risk of unwanted vendor lock-in. This allows developers, vendors and users to focus more on higher level capabilities and therefore less on reinventing common aspects and features of their APIs and interface modules. Coupling standards and software innovation is therefore crucial to economical cloud development at this stage.
  14. 14. Alan Sill, TTU September 24, 2013 14 Why Pursue Standards as a Means to Interoperability? •Risk reduction: •Lessen risk of dead-end product design and orphan components. •Lessen or remove risk of vendor service lock-in. •Mitigate reusability barriers for software and data access. •Provide best-of-breed development and methods. •Mix-and-match for input & output of processing steps. •Allow innovation/competition at more interesting layers and development of better internal features. •Facilitate interoperation with other provider software services, components and infrastructures.
  15. 15. Alan Sill, TTU September 24, 2013 Business Benefits • Having an organized set of acceptance criteria can improve business value to members in the following ways: • For VENDORS, ensures that their product passes test acceptance conditions, leading to fewer customer complaints in the field. • For PROVIDERS, ensures that products they are hosting are well-behaved, with fewer unexpected error or service failure conditions. • For USERS, ensures that the products and services they use will be interoperable with the API, framework or standard they are using as the basis for their purchased services, • Pre-purchase specification for products and services can be done on a rational basis. • Productivity of programmer teams is maximized by having well-defined workflows for scenario and unit testing. 15
  16. 16. Alan Sill, TTU September 24, 2013 16 OGF Cooperative Agreements In Place as of Late 2013 OGF and DMTF: • OGF published the the OCCI Core, Infrastructure and HTTP Rendering specifications as GFD.183, 184 and 185, and is working on a JSON rendering. We created a joint work register with DMTF on OCCI and CIMI in 2011 and continue to cooperate with them on implementation and testing of the CIMI specification. DMTF has also joined OGF in the Cloud Plugfests. OGF and ISO: • OGF has an ongoing Category A liaison with ISO JTC1 SC38 on Cloud Computing and is working with ISO on joint activities. OGF and SNIA (CDMI): • OGF has cooperative agreement w/SNIA with respect to CDMI and has co-hosted 10 Cloud Plugfests so far (series continues).
  17. 17. Alan Sill, TTU September 24, 2013 17 OGF and ETSI: • Cooperative MoU in place since 2012. OGF contributing to ETSI CSC effort. ETSI joined the Cloud Plugfest testing as a co-sponsor in 2012 and continues to support this series. OGF and ITU-T: • OGF has an ongoing formal liaison with ITU-T JCA Cloud. OGF and TM Forum: • MoU in place since 2005, renewed 2012. OGF contributed to TM Forum cross-SDO work on on “End-to-End Management of Cloud Service Agreements”, including SLAs, ongoing. OGF and CSA: • Cooperative agreement between OGF & CSA since formation. OGF and IEEE: • OGF co-sponsored IEEE Cloudcom conferences in 2011 (Athens) and 2012 (Taipei); ongoing engagements. OGF Cooperative Agreements In Place as of Late 2013
  18. 18. Alan Sill, TTU September 24, 2013 18 OGF and XSEDE: • OGF actively supports the standards process that underlies XSEDE infrastructure and many of its ongoing projects. OGF and EGI: • OGF works closely with EGI and associated projects; recently co-located OGF39 + Cloud Interoperability Week at EGI TF. OGF and OCEAN: • OCEAN joined Cloud Plugfest series in 2013 as a co-sponsor and as one of the primary sponsors of the upcoming September 2013 Cloud Interoperability Week. OGF and OW2: • OW2 co-sponsored Sep. 2013 Cloud Interoperability Week & Cloud Plugfest series and supports associated cloud projects. OGF Cooperative Agreements In Place as of Late 2013
  19. 19. Alan Sill, TTU September 24, 2013 19 OGF and NIST: • OGF contributed several talks and input to US National Institute of Standards and Technology Cloud Standards Roadmap in 2010-2013 timeframe; AFS co-chairs SAJACC working group. OGF and European Commission FP7/H2020 projects: • OGF technologies underpin the approaches of many successfully funded FP-series projects and expects to participate in supporting the Horizon 2020 call. OGF and Cloud Plugfest series: • OGF was one of the co-founding members of the ongoing Cloud Plugfest developer-oriented testing series and co- sponsored the recent Cloud Interoperability Week. OGF provides substantial infrastructure to the ongoing series. OGF Cooperative Agreements In Place as of Late 2013
  20. 20. Alan Sill, TTU September 24, 2013 NIST SAJACC Public Process 20
  21. 21. Alan Sill, TTU September 24, 2013 European Union “Standards & Interoperability for e-Infrastructure Implementation” (SIENA) Initiative SIENA  (2010-­‐2012)  is  a  Support  Ac8on  funded  by  the  European  Commission  under  FP7  (2007-­‐13)  CapaciDes  programme. A  coordinated  effort... ...towards  the  delivery  of  a  future              e-­‐ Infrastructures  Roadmap.... ...aligned  with  the  needs  of  European  and  na=onal   ini=a=ves  and  the  evolving  world. IEG  –  Industry  Expert   Group Partners Interna8onal  Experts  REB  –  Roadmap   Editorial  Board SLG  –  Special  Liaison   Group Standards  Development  Organisa8ons  (SDOs) Distributed  Compu8ng  Infrastructure  (DCI)  Ini8a8ves Consolidated  Cloud  Compu8ng  Workshops Final Report: June 2012 21
  22. 22. Alan Sill, TTU September 24, 2013 22
  23. 23. Alan Sill, TTU September 24, 2013 Cloud Plugfest Developer Series: 24 Continuing series co-sponsored by OGF, DMTF, SNIA, OASIS, ETSI, OCEAN and OW2! 9 previous events held so far! Developer-oriented in- person standards and software testing series OPEN TO ALL! Easy to get involved and join in events as developers or project researchers! Cloud Plugfest 10 just completed! Cloud Interoperability Week Sep. 2013! http://
  24. 24. Alan Sill, TTU September 24, 2013 ETSI Tools Used in Cloud Plugfests: 25
  25. 25. Exploration
  26. 26. Proof that iPhones are used in space: Link: Voyager - The Interstellar Mission
  27. 27. Alan Sill, TTU September 24, 2013 Conclusions • OGF actively engages with partners and participants throughout the international arena through an open forum with open processes to promote best practices and standards in advanced distributed computing. • OGF and its associated projects occupy an important role in standards and software development with significant uptake in advanced distributed computing, including cloud, grid, networking and large-scale data processing, transfer and handling through innovative cooperation with many partners. • TTU is leveraging these standards to support a wide variety of flexible architectures for advanced scientific and business uses through the NSF Cloud and Autonomic Computing Center and is actively seeking international collaborators to extend this work more comprehensively through the cloud. 28