Cloud Standards in the Real World: Cloud Standards Testing for Developers

982 views

Published on

Learn about standards studied in the US National Science Foundation Cloud and Autonomic Computing Industry/University Cooperative Research Center Cloud Standards Testing Lab and how you can get involved to extend the successes from these results in your own cloud software settings. Presented at the O'Reilly OSCON 2014 Open Cloud Day.

Video available at https://www.youtube.com/watch?v=eD2h0SqC7tY

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
982
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Cloud Standards in the Real World: Cloud Standards Testing for Developers

  1. 1. Open Cloud Standards
 In The Real World:
 Cloud Standards Testing For Developers A Project Of the National Science Foundation Cloud and Autonomic Computing
 Industry/University Cooperative Research Center At Texas Tech University Alan Sill, Ph.D Site Director July 21, 2014OSCON Open Cloud Day
  2. 2. First, some preliminaries… What it often looks like when developers encounter standards committees. OSCON 2014 Open Cloud Day July 21, 2014
  3. 3. What it ought to look like: (Taken from an actual Cloud Plugfest.) OSCON 2014 Open Cloud Day July 21, 2014
  4. 4. A New Research Effort OSCON 2014 Open Cloud Day July 21, 2014
  5. 5. A New Research Effort OSCON 2014 Open Cloud Day July 21, 2014
  6. 6. NSF CAC Current Participants: OSCON 2014 Open Cloud Day July 21, 2014 CAC Industry Members: Technical Partnerships: Other CAC University Sites: Others to come! Texas Tech University Other CAC Sites … (Pending)(Existing)
  7. 7. New NSF Award —
 April 2014
  8. 8. Background: Large-Scale Academic Computing OSCON 2014 Open Cloud Day July 21, 2014 In Top500! ! 3rd in Texas and Big 12! ! In Top 100 academic institutions in the world ! Supports high-quality academic research across many fields of science, business data analysis
 and engineering. TTU High Performance Computing Center:
  9. 9. Background: Big Data OSCON 2014 Open Cloud Day July 21, 2014 2014:! Phones: 100+ Gigabytes! Science and Business: 
 100s to 1000s of Petabytes
  10. 10. Experience:
 
 Related CyberInfrastructure Projects
  11. 11. Experience In Global-Scale Computing OSCON 2014 Open Cloud Day July 21, 2014
  12. 12. The Role of Standards for Risk Reduction and Inter-operation in XSEDE XSEDE: The Next Generation of US Supercomputing Infrastructure Cloud and grid standards now power some of the largest academic supercomputing infrastructures in the world!
  13. 13. Blacklight   Shared  Memory   4k  Xeon  cores ! Darter   24k  cores   ! Nautilus   Visualization   Data  Analytics   ! Keeneland    CPU/GPGPU Stampede   460K  cores   w.  Xeon  Phi   >1000  users   Upgrade  in  2015 Wrangler   Date  Analytics Trestles   IO-­‐intensive   10k  cores   160  GB  SSD/Flash   ! Gordon       Data  intensive   64  TB  memory   300  TB  Flash  Mem Open  Science  Grid   High  throughput   124  sites Blue  Waters   Leadership  Class SuperMIC   380  nodes  –  1PF   (Ivy  bridge,  Xeon  Phi,  GPU) Over  13  million  service  units/day  typically   delivered  as  of  2014  across  all  XSEDE   supercomputing  sites  (about  3  million  core  hours/ day),  totaling  about  1.6  billion  core  hours  per  year Yellowstone   Geosciences US National Cyberinfrastructure Promote an open, robust, collaborative, and innovative ecosystem Adopt, create and disseminate knowledge Extend   the impact 
 of cyber- infrastructurePrepare the current and next generation Provide technical expertise and support services Collaborate with other CI groups and projects NSF Supercomputing
 Resources OSCON 2014 Open Cloud Day July 21, 2014 FutureGrid*   CS  testbed Maverick   Visualization   Data  Analytics Comet   “Long  Tail  Science”   47k  cores/2  PF   High  throughput ACI-­‐REF   Campus  sharing Grids Credit: Irene Qualters, US National Science Foundation
  14. 14. XSEDE Services Layer: Simple services combined in useful ways –Resource  Namespace  Service  1.1   –OGSA  Basic  ExecuKon  Service   –OGSA  WSRF  BP  –  metadata  and  noKficaKon   –OGSA-­‐ByteIO   –GridFTP   –JSDL,  BES,  BES  HPC  Profile   –WS  Trust  Secure  Token  Services   –WSI  BSP  for  transport  of  credenKals   –…  (more  than  we  have  room  to  cover  here) Examples – (not a complete list) Bottom line: XSEDE represents a demonstration of the effective use of standards 
 to provide interoperability within large-scale supercomputing cyberinfrastructure. Used between supercomputers and all the way to university campuses!
  15. 15. Distributed Across 124 Sites Open Science Grid currently consists of over 124 geographical sites, operating on a wide variety of computing systems 600k - 800k jobs/day!
  16. 16. Virtual Organizations, July 18th 2011 Science VOs on the Open Science Grid • Astrophysics • Biochemistry • Bioinformatics   • Earthquake  Engineering • Genetics • Gravitational-­‐wave  physics   • Mathematics • Nanotechnology • Nuclear  and  particle  physics … and many others!
  17. 17. www.egi.euEGI-InSPIRE RI-261323 www.egi.euEGI-­‐InSPIRE  RI-­‐261323 XSEDE/EGI BoF - XSEDE 2013 EGI international presence numbers Storage Value  (yearly  increase) Disk  (PB)    235  PB  (+69%) Tape  (PB)    176  PB  (+32%) Value  (yearly  increase) CPU  cores 361,300  across  53  countries   (1.44  M  job/day) Standards-based international collaboration
  18. 18. EGI Federated Cloud: a successful standards-based international federated infrastructure Credit: David Wallom Chair EGI Federated Cloud Task Force Members •70 individuals •40 institutions •13 countries Stakeholders •23 Resource Providers •10 Technology Providers •7 User Communities •4 Liaisons Technologies •OpenStack •OpenNebula •StratusLab •CloudStack (in evaluation) •Synnefo •WNoDeS BSC CNRS LMU OeRC Masaryk TUD IFAE Cyfronet 100%IT CESNET RADICAL SRCE DANTE FZJ GRNET GWDG STFC SARA KTH INFNFCTSG EGI.eu Imperial CESGACETA IFCA IGI IPHC IN2P3 SZTAKI IISAS SixSq Standards •OCCI (control) •OVF (images) •X.509 (authN) •CDMI (storage - under development) (Updated July 2014) An international standards- based
 federated cloud infrastructure:" 
 In production, 
 in use and growing!
  19. 19. www.egi.euEGI-InSPIRE RI-261323 Federated Cloud architecture Open to new members: Join as user, or as an IaaS/PaaS/SaaS service provider: http://go.egi.eu/cloud Cloud  hypervisor
 (e.g.  OpenStack,  OpenNebula,   EmotiveCloud,  Okeanos…) Cloud  site
 academic/commercial Domain  specific  services  in   Virtual  Machine  Images FedCloud  User  interfaces Standards  used  to  enable  federation   • OCCI:  VM  Image  management   • OVF:  VM  Image  format   • X509:  Authentication   • (CDMI:  Storage) FedCloud   Operation   interfaces • Information  system  (BDII)   • Monitoring  (SAM)   • Accounting  (APEL)   • AAI  (Perun) Virtual  organisations • GLUE2:  Resource  discovery   and  Description   • Others  in  development   Federation  monitoring
  20. 20. Examples of CAC@TTU Involvement In Cloud Standards Testing
  21. 21. US NIST SAJACC Working Group
 Alan Sill, TTU and Eugene Luster, R2AD, co-chairs NIST Cloud Computing and Mobility
 Forum and Workshop March 25, 2014 SAJACC Phase II Use Case Definition and Testing Project for Cloud Standards,APIs and Cloud Software Product Implementations
  22. 22. http://collaborate.nist.gov/twiki-cloud-computing/bin/view/CloudComputing/SAJACC NIST SAJACC Public Process OSCON 2014 Open Cloud Day July 21, 2014
  23. 23. European Commission“Standards & Interoperability for 
 e-Infrastructure Implementation”Initiative SIENA  (2010-­‐2012)  was  a  Support  Ac*on  funded  by  the  European   Commission  under  FP7  (2007-­‐13)  CapaciTes  programme.   This   project   created   the   ongoing   “CloudScape”   conference   series   and  led  to  other  related  projects,  such  as  CloudWatchHub  and  the   CloudScout   effort   aimed   at   enhancing   the   adopTon   of   cloud   methods  by  SMEs. A  coordinated  effort... ...towards  the   delivery  of  a  future
 e-­‐Infrastructures   Roadmap.... ...aligned  with  the   needs  of  European   and  naVonal   iniVaVves  and  the   evolving  world. IEG  –  Industry  Expert  Group Partners Interna*onal  Experts  REB  –  Roadmap  Editorial   Board SLG  –  Special  Liaison  Group Standards  Development  Organisa*ons  (SDOs) Distributed  Compu*ng  Infrastructure  (DCI)  Ini*a*ves Consolidated  Cloud  Compu*ng  Workshops 23
  24. 24. Open-access academically hosted Cloud Interoperability Testbed project
  25. 25. CAC@TTU Project Areas
  26. 26. Target Cloud Standards-Related Organizations: • It is often said that there are “too many standards organizations”. 
 This is a lot like saying there is “too much software”. • Each has its own area of specialty, its own contributor base, and its 
 own method of funding to develop its work products. • CAC will study products and effectiveness of each of these organizations. OSCON 2014 Open Cloud Day July 21, 2014 …
  27. 27. Initial CAC@TTU Project Areas 1. Product and Standards Testing • Cloud Performance Testbed • Cloud Standards Testbed • Cloud Interoperability Testbed • Cloud Security Testbed <— (Future) 2. Design Labs • Storage Design and Testing Lab • Network Design and Testing Lab 3. Developer Events • Cloud Plugfest Series • Participation in technical partner events • Organization of and participation in conferences OSCON 2014 Open Cloud Day July 21, 2014 p CAC@TTU is new! n More coming…
  28. 28. • Cloud Standards Testbed! • Cloud Performance Testbed! • Cloud Interoperability Testbed! • Cloud Tester Benchmark Suite Core Technology Efforts * * (In cooperation with The Aerospace Corporation and other CAC partners)
  29. 29. Continuing series… ! Oriented towards REAL DEVELOPMENT ! Past events
 co-sponsored by many open source and standards-related organizations including 
 OGF, DMTF, SNIA, OASIS, ETSI, OCEAN and OW2 Developer-oriented in- person standards and software testing series Easy to get involved and join in events as developers or project researchers Cloud Plugfest 11 just completed! http://cloudplugfest.org Cloud Plugfest Developer Series: OSCON 2014 Open Cloud Day July 21, 2014 More events in planning pipeline.
  30. 30. CAC
 Cloud Standards Testing and Implementation Projects
  31. 31. Testing Cloud-Oriented Use Cases § Builds on results of publicly accessible documents and forums for general interoperability testing, in addition to internal ones that may be set up by companies and US agencies. § Up to now these have mostly been approached piecemeal. § CAC is carrying out an organized project within the center to extend these techniques into tools that can be used for formal support of cloud conformance, interoperability and requirements acceptance testing on an ongoing basis. § This project builds on previous public efforts such as SAJACC. § Our plan is to work with and contribute to public processes in open settings while creating conditions in which our expertise and value can be applied for center members to cloud standards and product testing. OSCON 2014 Open Cloud Day July 21, 2014
  32. 32. Current Standards Being Tested § Cloud Data Management Interface (CDMI) from the Storage Networking Industry Association (SNIA). (Reference implementation available.) § Open Cloud Computing Interface (OCCI) from the Open Grid Forum (OGF). (Many implementations available.) § Cloud Infrastructure Management Interface (CIMI) and Open Virtualization Format (OVF) from the Distributed Management Task Force (DMTF). (CIMI implementations just becoming available; OVF implementations well established.) § Topology and Orchestration Specification for Cloud Applications (TOSCA) from the Organization for the Advancement of Structured Information Standards (OASIS). (Several implementations available.) § Others under active evaluation. OSCON 2014 Open Cloud Day July 21, 2014
  33. 33. Texas Tech Leadership Team Alan Sill, Ph.D Site Director Yong Chen, Ph.D Associate Site Director Ravi Vadapalli, Ph.D Associate Site Director
  34. 34. Experienced university-based researchers carrying out projects with real-world applicability that draw on decades of experience and the best, newest techniques. Publication oriented, but available for practical member-driven projects. Full university support from TTU and other participating CAC university sites. OSCON 2014 Open Cloud Day July 21, 2014 v COMPUTER SCIENCE Rattikorn Hewett, Ph.D. Chairperson in the Department of Computer Science Areas of expertise » Security and privacy in cloud environments » Intelligent data management & understanding » Automated web service composition Sunho Lim, Ph.D. Assistant professor in the Department of Computer Science Areas of expertise » Wireless networks and mobile computing » Mobile data management » Mobile software » Network security Mohan Sridharan, Ph.D. Assistant professor in the Department of Computer Science Areas of expertise » Learning and estimation in Big Data domains » Stochastic machine learning » Robotics and AI Noe Lopez-Benitez, Ph.D. Associate professor in the Department of Computer Science Areas of expertise » Cloud computing » Parallel and distributed computing » High-performance computing Yong Chen, Ph.D. Assistant professor in the Department of Computer Science Areas of expertise » Cloud computing storage systems » Data management and data centers » Cloud programming and software environments » Cloud platform architecture and virtualization Eunseog Youn, Ph.D. Assistant professor in the Department of Computer Science Areas of expertise » Large-scale machine learning » Feature selection » Text mining » Bioinformatics FACULTY MEMBERSHIGH PERFORMANCE COMPUTING CENTER SCHOOL OF LAW Victoria Sutton, Ph.D. in the School of Law Areas of expertise » Biosecurity » Emerging technologies » Energy » Constitutional law Many University Researchers, Deep Expertise
  35. 35. INDUSTRIAL ENGINEERING Timothy Matis, Ph.D. Associate professor in the Department of Industrial Engineering Areas of expertise » Operations research » Stochastic processes » Queueing theory » Ad-hoc communication networks » Conceptual learning theories and virtual learning environments Areas of expertise » Mixed-integer programming, theory and computation » Nonlinear programming, theory and computation » Operations research probabilistic methods » Applied management science MATHEMATICS & STATISTICS Philip W. Smith, Ph.D. Senior director, High Performance Computing Center Areas of expertise » Optimization » Design and implementation of parallel and serial codes » Code optimization » Numerical analysis and linear algebra MECHANICAL ENGINEERING Fazle Hussain, Ph.D. Professor in the Department of Mechanical Engineering Areas of expertise » Fluid Flow, turbulence, & transport phenomena » Microseismology and inverse scattering » Modeling cancer tissue growth and targeted drug delivery » Atmospheric boundary layers and wind turbine aerodynamics Siva Parameswarn, Ph.D. Professor in the Department of Mechanical Engineering Areas of expertise » Modeling HF oscillatory ventilators » Aerodynamics of ground vehicles » Wake development behind wind turbines PHYSICS Ismael Regis de Farias Jr., Ph.D. Associate professor in the Department of Industrial Engineering Alan Sill, Ph.D. Adjunct Professor in the Department of Physics Areas of expertise » High performance computing » Data-intensive scaled resource management » Large-scale distributed computing » Standards development Experienced university-based researchers carrying out projects with real-world applicability that draw on decades of experience and the best, newest techniques. Publication oriented, but available for practical member-driven projects. Full university support from TTU and other participating CAC university sites. OSCON 2014 Open Cloud Day July 21, 2014 Many University Researchers, Deep Expertise
  36. 36. FACULTY MEMBERS RAWLS COLLEGE OF BUSINESS Peter H. Westfall, Ph.D. Statistics in the Rawls College of Business Administration Zhangxi Lin, Ph.D. Associate professor in the Department of ISQS Areas of expertise » Advanced analytics » Business intelligence » Statistics » Biostatistics » Bioinformatics Areas of expertise » Big data analytics » Business intelligence » Information systems » Electronic commerce Mayukh Dass, Ph.D. Assistant Professor of Marketing in the Rawls College of Business Areas of expertise » Brand analytics » Business strategy analysis » Dynamic environment modeling » Network analysis PETROLEUM ENGINEERING Marshall Watson, Ph.D., P.E. Roy Butler Chair and chairman of the Bob L. Herd Department of Petroleum Engineering Mohamed Soliman, Ph.D., P.E. George P. Livermore Chair and Professor in the Bob L. Herd Dept. of Petroleum Engineering Assistant professor in the Bob L. Herd Department of Petroleum Engineering Ravi Vadapalli, Ph.D. Adjunct Professor in the Department of Petroleum Engineering Areas of expertise » Production optimization » Coal bed methane » Unconventional reservoir analysis Areas of expertise » Fracturing » Reservoir engineering » Well test analysis » Conformance Areas of expertise » Reservoir simulation » Advanced well completion Areas of expertise » High performance computing » Reservoir modeling » Cancer radiotherapy informatics » Cloud and autonomic computing Many University Researchers, Deep Expertise Experienced university-based researchers carrying out projects with real-world applicability that draw on decades of experience and the best, newest techniques. Publication oriented, but available for practical member-driven projects. Full university support from TTU and other participating CAC university sites. OSCON 2014 Open Cloud Day July 21, 2014
  37. 37. Links For Further Information and To Join: § NSF Cloud and Autonomic Computing Center main site: 
 http://nsfcac.org § CAC@TTU information and membership materials:
 http://www.depts.ttu.edu/cac/ § Cloud Plugfest developer series: http://cloudplugfest.org § NSF I/UCRC main site: http://www.nsf.gov/eng/iip/iucrc/ § Cloud standards organization compilation: 
 http://cloud-standards.org § NSF CAC@TTU contact email: cac.info@ttu.edu OSCON 2014 Open Cloud Day July 21, 2014

×