The UberCloud - From Project to Product - From HPC Experiment to HPC Marketplace - From HPC Shop to HPC Shopping Mall

570 views
496 views

Published on

The UberCloud online marketplace for engineers and scientists to discover, try, and buy compute power on demand, in the cloud. Starting with free experiments in the cloud, including application software, cloud hardware, and expertise. Learning by doing how to use your application in the cloud.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
570
On SlideShare
0
From Embeds
0
Number of Embeds
12
Actions
Shares
0
Downloads
7
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

The UberCloud - From Project to Product - From HPC Experiment to HPC Marketplace - From HPC Shop to HPC Shopping Mall

  1. 1. The UberCloud From Project to Product From HPC Experiment to HPC Marketplace From HPC Shop to HPC Shopping Mall Wolfgang Gentzsch President,The UberCloud BurakYenier CEO,The UberCloud HPC 2014 , Cetraro, July 7 – 11, 2014
  2. 2. The UberCloud From Project to Product From HPC Experiment to HPC Marketplace From HPC Shop to HPC Shopping Mall Wolfgang Gentzsch President,The UberCloud BurakYenier CEO,The UberCloud Product innovation and scientific insight require computing <= HPC 2014 , Cetraro, July 7 – 11, 2014
  3. 3. Summary: UberCloud Progress  Traction: 1,500 registered orgs, 72 countries, 155 experiment teams exploring Computing as a Service  Visible: 60+ articles; 40+ trade shows; prestigious 2013 HPCwire Readers’ ChoiceAward  Powerful sponsors: Intel,Autodesk, Bull, IDC,ANSYS; talking to 10 more.  Powerful participant: 4 ofTop 5 CAE ISVs (total 80+); 100+ sw/hw providers; hundreds of end-users, 600+ renowned experts  Compendium I & II: 25 + 17 best case studies from Round 1 – 5 HPC Experiment sponsored by Intel, over 1,000 downloads  Hired LindaTreiman (from Bright) to take care of our providers and sponsors  Container technology and run time environment  UberCloud Marketplace andAppStore 3
  4. 4. Engineers & scientists computing tools: workstations 3 options to use technical compute power , servers, and clouds
  5. 5. Benefits of HPC in the Cloud Continue using your workstation for your daily design, and use Cloud resources with additional benefits:  An HPC system at your finger tip, on demand  Pay per use (no CAPital EXpenditure)  Scaling resources up and down (business flexibility)  Low risk by working with multiple cloud providers.
  6. 6. The challenges  Workstation: slow, limited capacity  HPC server: expensive (TCO!), complex  HPC in the Cloud: security, licensing, data transfer, expertise, and …  Very crowded cloud services market, difficult to find your ideal service
  7. 7. It all started June 2012 with the free voluntary UberCloud Experiments HPC as a Service, on demand, in a team experiment For SMBs and their engineering applications to explore the end-to-end process of using remote computing resources, as a service, on demand, at your finger tip, and learning how to resolve the roadblocks.
  8. 8. How does the Experiment work?  End-User registers  SoftwareVendor joins  We select a Team Expert  Matching a Resource Provider  152 UberCloud Experiments so far  42 case studies in Compendium I & II  Assigning an UberCloud mentor  Now, the team is ready to go  Finally, writing the Case Study
  9. 9. 22 StepsTowards a successful project Step 1: define end-user project  1.1: TE & EU fill out "Project definition" docu  1.2: UC assigns SP based on "Project definition" docu  1.3: UC +TM assign RP based on "Project definition" docu  1.4: TE calls for a kick-off meeting over Skype via Doodle  1.5: RP fills out "Computing resources" docu  1.6: SP fills out "Software resources" docu  1.7: If custom code, EU fills out "Software resources" docu  1.8 TE +TM review UC Exhibit, consider additional services EU = end user, SP = software provider, RP = resource provider,TE = team expert, TM = team mentor, UC UberCloud
  10. 10. 22 StepsTowards a successful project Step 2 & 3: resources & execution Step 2: Contact the resources, set up the project environment  2.1: TE gets resources using "Computing resources" docu  2.2: TE & RP set up software using "Software resources" docu  2.3: TE & RP set up EU code using "Software resources" docu  2.4: TE & RP configure project environment  2.5: TE performs a trial run Step 3: Initiate project execution on cloud resources  3.1: TE & EU upload data to the project environment  3.2: TE & RP queue the job(s) for the project EU = end user, SP = software provider, RP = resource provider,TE = team expert, TM = team mentor, UC UberCloud
  11. 11. 22 StepsTowards a successful project Step 4-6: monitor, review, report Step 4: Monitor the project  4.1: TE monitors the job status  4.2: TE & EU re-set parameters between runs as needed  4.3: TE & RP performs post processing, such as remote viz Step 5: Review your results  5.1: TE makes results available to EU, if needed repeats Step 2-5  5.2: TE & RP remove EU data from project environment Step 6: Document your findings  6.1: TE initiates docu "Template for UC Experiment Uses Cases"  6.2: TE requests team to contribute to and review the docu EU = end user, SP = software provider, RP = resource provider,TE = team expert, TM = team mentor, UC UberCloud
  12. 12. Step by Step process Basecamp project management platform for each team
  13. 13. The UberCloud HPC Experiments Started July 2012, 1500 participants, 72 countries Example: AmazonAWS in the UberCloud:  Team 2:  Team 20:  Team 30:  Team 40:  Team 65:  Team 70:  Team 116:  Team 142:  Team 147: 13 Simulation of a Multi-resonant Antenna System Turbo-machinery Application Benchmarks HeatTransfer Use Case Simulation of Spatial Hearing Weather Research with WRF Next Generation Sequencing Data Analysis Quantitative Finance Historical Data Modeling VirtualTesting of Severe Service ControlValve Compressor Map Generation Using Cloud-Based CFD
  14. 14. The UberCloud HPC Experiments Started July 2012, 1500 participants, 72 countries Example: Bull extreme factory in the UberCloud:  Team 5:  Team 8:  Team 32:  Team 52:  Team 85:  Team 89:  Team 120: 14 2-phase Flow Simulation of a Separation Column Flash Dryer with Hot Gas to EvaporateWater from a Solid 2-phase flow simulation of a separation columns Simulations of Blow-off in Combustion Systems Combustion simulations of power plant equipment Simulations of Enzyme-Substrate reactions Simulation of water flow around self-propelled ship
  15. 15. © 2013 ANSYS, Inc. July 16, 201415 Some Lessons Learned - UberCloud HPC Experiment Team 8: Flash Dryer Simulation (ANSYS Fluent) Simulation throughput criterion was met ‼ Remote visualization solution required ‼ Time for downloading results ‼ IP concern Team 9: Irrigation Simulation (ANSYS CFX) Timely, high fidelity results were obtained ‼ Windows above Linux preferred ‼ HPC workshop services for SMEs requested Ability to conduct parametric simulations ‼ Sufficient number of licenses needed ‼ Remote visualization solution required ‼ Disappointing hardware performance results Team 34: Wind Turbine Simulation (ANSYS Fluent) Source: The UberCloud HPC Experiment: Compendium of Case Studies
  16. 16. © 2013 ANSYS, Inc. July 16, 201416 Some Lessons Learned - UberCloud HPC Experiment Team 36: IC-Engine Simulation (ANSYS Fluent) Smooth setup of environment and sw ‼ Appropriate cloud licensing required ‼ Network bandwidth not good for graphics ‼ Customized sw needs to be recompiled Team 54: Pool Plant Simulation (ANSYS CFX) Ability to easily burst into the Cloud Accelerated file transfer and 3D graphics ‼ Cost of the commercial CFD licenses Ease of use Good remote visualization ‼ File uploading time ‼ Stress test with multiple users required Team 56: Axial Fan Simulation (ANSYS Fluent) Source: The UberCloud HPC Experiment: Compendium of Case Studies
  17. 17. Team 1: Heavy DutyABAQUS Structural Analysis in the Cloud TheTeam:  Frank Ding, is the EngineeringAnalysis and Computing Manager at Simpson Strong-Tie in Northern California.The end user problem space…..  Matt Dunbar, is now the ChiefArchitect and CAE technical specialist at Simulia Dassault Systems, in Rhode Island on the East Coast. He represents the application level expertise in this experiment.  Steve Hebert, is one of the founders and CEO of Nimbix, located inTexas, which in this team is the provider of cloud-based HPCinfrastructure and applications hosting  Rob Sherrard, is the other co-founder of Nimbix andVP of Service Delivery.  Sharan Kalwani, HPC Segment Architect with Intel Corporation and in this project is the overall Subject Matter Expert,located in Michigan (Midwest).
  18. 18. Team 1:The problem to be solved  The Use Case:  ABAQUS/Explicit and ABAQUS/Standard are the major applications  HPC cluster at Simpson Strong-Tie is modest, 32 cores of Intel x86-based gear.  Cloud bursting is critical.  Also challenging is the issue of sudden large data transfers  Need to perform visualization ensuring design simulation is proceeding correctly  Workflow  Pre-processing happens on end user’s workstation to prepare the CAE model  Files transferred to HPC cloud data staging area using a secured FTP process  Submit the job through (Nimbix.net) web portal  Result files can be transferred back for post-processing,  or the post-processing can be done using remote desktop tool like HP RGS on the HPC provider’s visualization node.
  19. 19. Team 1: Challenges!  A weekly schedule – was not the first challenge!  Needed a fast interconnect (e.g. Infiniband) which was not available.  Solved with “fat” nodes, as this cluster is a sandbox for testing the cloud workflow, the actual inter-connect performance of this 12 core cluster was not a concern.  The second challenge was to address the need for simple and secure file storage and transfer. Accomplished very quickly using GLOBUS technology.These days cloud based storage is mature and ready for prime time HPC, especially in the CAE arena.  The third challenge was now to push the limits and stream several jobs simultaneously to the remote HPC cloud resource.This provided solid evidence that “bursting” was indeed feasible.To the whole team’s surprise it worked admirably and had no impact whatsoever overall.  The fourth and final challenge now became perhaps the most critical which was the end user perception and acceptance of the cloud as a smooth part of the workflow.  Remote visualization was necessary to see if the simulation results (left remotely in the cloud)
  20. 20. Team 1:What the end user saw…..  With right tuning, useful remote visualization!
  21. 21. Team 1:What did we learn?  Benefits:  Clearly established - HPC cloud model can indeed be made to work.  Recommendations:  A few key necessary factors emerged:  Result file transfers: most CAE result files easily over several gigabytes, a minimum of 2-4 MB/sec sustained and delivered bandwidth is necessary  The same applies when doing remote visualizations, in this case, 4 MB/sec is the threshold Latency is also a key concern.  Beyond the Cloud service provider, a network savvy ISP is perhaps a necessary part of the team of infrastructure in order to deliver robust and production like HPC cloud  Remote visualization provides a convenient collaboration platform for a CAE analyst to access the analysis results any where he has the need, but it requires a secure “behind the firewall” remote workspace
  22. 22. Team 2: Simulating new probe design for a medical device HPC Expert: End User: wanted to stay anonymous Credits from:
  23. 23. Team 70 Case Study: Next Generation Sequencing DataAnalysis  MEET TEAM 70:  End User -Thomas Dyar, Senior Genomics Data Scientist, Betty Diegel, Senior Software Engineer, medical devices company  Software Provider - Brian O'Connor, CEO Nimbus Inform.. Cloud services for workflows utilizing SeqWare  Resource Provider - AmazonWeb Services  HPC Cloud Experts - Cycle Computing
  24. 24. Team 142 Case Study:Virtual testing of severe service control valve  MEET TEAM 142:  End User – Mark Lobo, Lobo Engineering;  Software Provider – Derrek Cooper,Autodesk CFD 360  Resource Provider - AmazonWeb Services  HPC Cloud Experts – Jon den Hartog and Heath HoughtonAutodesk
  25. 25. Challenges with the experiments  HPC is complex; at times it requires multiple experts  Reaching out to industry end-users  No standards: access and usage of hw & sw providers are different, some are complex  Lack of automation: Currently the end-to-end process of the HPC experiment is manual (intentionally).  Time delays: vacation, conferences, and everybody has a day job (busy!)  Barriers: Complexity, data transfer, security, IP, software licenses, performance, interoperability… AND: we learn a lot . . . .
  26. 26. Bumps on the road  Time delays:Vacation times in July/August and December  No standards: Access and usage processes of hw & sw providers are different, some complex  Hands-on: Process automation at providers vary greatly.  Lack of automation: Currently the end-to-end process of the HPC experiment is manual (intentionally).  Participants spent relatively small portion of their time, some are responsive, others are not: it is not their day job!  Getting regular updates fromTeam Experts is a challenge because this is not their day job !
  27. 27. Building a marketplace demands building an ecosystem UC Market Place App store Comm unity Start: Rough idea Mar Com ß Pro duct Techn ology Exper iment Exhib ition 06/12 HPC Cetraro 09/12 01/13 01/13 01/14 01/14 03/14 06/14 workflow impact
  28. 28. Problem: today’s crowded and ineffective cloud ‘market’ Supply Cloud providers ISVs Consultants Trainers Demand Engineers Scientists Data analysts Experts . . . . . Complexity Data Transfer SecurityLicensing Uncertain Cost Roadblocks
  29. 29. Solution: The UberCloud Marketplace Supply Cloud providers ISVs Consultants Trainers … Demand Engineers Scientists Data analysts Experts UberCloud Marketplace
  30. 30. Solution: The UberCloud Marketplace UberCloud Marketplace for 20+ million engineers and scientists and their service providers to discover, try, buy, and sell computing time, storage, software and expertise on demand
  31. 31. Announcement at HPC Cetraro
  32. 32. Technology solution: StandardCloud run-time environment  Building thin, light-weight run-time environment (RTE) on top of Linux kernel features and open source tools, which  provides a standard platform across distributed in-house, grid, and cloud resources  facilitates access to all kinds of resources (workstations, servers, and private, hybrid, and public clouds)  moving portable, stackable units including end-users app, data, tools seamlessly btwn in-house and external resources  enables portability across different in-house and external resources (federation)  reducing / removing many of the cloud challenges 32
  33. 33. Builder Launcher Controller ISV DataTools Stackable units with tools (ex: encryption), ISV application codes (ex: OpenFOAM). Just add your own codes and data. Run anywhere with UberCloud Run Time. Scale up or down the compute power as needed. Collect granular usage data, logs. Monitor, alert, report. Any Workstation Any Cluster Any Cloud Run Time Run Time Run Time Build once, run anywhere
  34. 34. Portable Units are like containers  Standard software units (with user’s app, data, tools etc.) can be moved seamlessly across any set of resources. Units are  stackable and portable,  built from a base unit with standard functionality (security, encryption, compression, monitoring, data transfer, etc)  extended by the ISV’s software as next layer,  top layer is the end-users configuration and data. 34
  35. 35. Next Steps: Reducing / Removing Cloud Challenges Challenge *) Addressed today With UberCloud **) Portability low high Security medium high Software Licenses low medium DataTransfer low medium Compliance low medium Standardization low high Cost & ROITransparency low high Resource Availability medium high Transparency of Market low high Cloud Computing Expertise low medium *) Cloud challenges are addressed low, or medium, or high **)When UberCloud is fully developed one year from now
  36. 36. It’s your turn now   Download 2013 Compendium of case studies from HPCwire  Download 2014 Compendium of case studies  Register atTheUberCloud.com  Try the UberCloud Marketplace with $1 voucher and you get  NOW NOW
  37. 37. The UberCloud Community and Marketplace ThankYou ! Register free at http://www.TheUberCloud.com

×