Cloud software engineering

2,972 views

Published on

Introduces the characteristics of clouds and discusses the challenges that these pose for engineering scientific applications on the cloud. Key challenges are programming models, developing PaaS interfaces for high performance/high throughput computing and developing an SDE for cloud programmng.

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,972
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
118
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Cloud software engineering

  1. 1. Challenges for cloud software engineering Ian Sommerville St Andrews University Challenges for Cloud Software Engineering, 2012 Slide 1
  2. 2. Why is cloud software engineeringdifferent – or is it? What needs to be done to make cloudsoftware engineering easier for theresearch community? Challenges for Cloud Software Engineering, 2012 Slide 2
  3. 3. Public, private, community clouds Challenges for Cloud Software Engineering, 2012 Slide 3
  4. 4. There will be a number of public cloudproviders offering different capabilities atdifferent prices at different times Public cloud computing prices willcontinue to fall and it will be the mosteconomical approach Challenges for Cloud Software Engineering, 2012 Slide 4
  5. 5. Private (institutional) clouds for scienceresearch are not really viable Community clouds make sense for data-intensive computing if they are locatedclose to the sources of the data Community clouds hosting standardsoftware may be a useful approach –possibly hosted on public clouds Challenges for Cloud Software Engineering, 2012 Slide 5
  6. 6. What makes the cloud different (and challenging)? Challenges for Cloud Software Engineering, 2012 Slide 6
  7. 7. Scale Thousands of nodes that may be distributed across the world Challenges for Cloud Software Engineering, 2012 Slide 7
  8. 8. Elasticity No fixed hardware architecture Scale out and scale in Challenges for Cloud Software Engineering, 2012 Slide 8
  9. 9. Pay as you go pricing Variable costs for applications Payment according to resources used Challenges for Cloud Software Engineering, 2012 Slide 9
  10. 10. Cloud services Challenges for Cloud Software Engineering, 2012 Slide 10
  11. 11. Infrastructure as a service Challenges for Cloud Software Engineering, 2012 Slide 11
  12. 12. Instead of buying and running physical equipment (computers, storage, etc.), you ‘rent’ these as required from a cloud provider You ship your programs and data to a remote data centre and run them there rather than locally The familiar Grid computing model Challenges for Cloud Software Engineering, 2012 Slide 12
  13. 13. Platform as a service Challenges for Cloud Software Engineering, 2012 Slide 13
  14. 14. The cloud provider makes a set of APIs available which you use in your program to interact with the platform Scaleability is transparent. Automatic scale out/scale in as demand changes Primarily geared to writing web-based applications Challenges for Cloud Software Engineering, 2012 Slide 14
  15. 15. Software as a service Challenges for Cloud Software Engineering, 2012 Slide 15
  16. 16. Software is shared by different users and accessed using a web browser Services are stateless, functional abstractions Multi-tenant databases for efficient operation Accessed via a service interface – SOAP or RESTful interface Challenges for Cloud Software Engineering, 2012 Slide 16
  17. 17. Levels of cloud usage Challenges for Cloud Software Engineering, 2012 Slide 17
  18. 18. Cloud hosting You use the cloud to provide server capability rather than incurring the capital costs of server purchase Applications need not be ‘cloud aware’ Fixed server infrastructure Can run existing codes in Fortran/C Challenges for Cloud Software Engineering, 2012 Slide 18
  19. 19. If server type and platform available, thenshould not require any applicationchanges Almost certainly poorer performancethan local execution. But makes sense for occasional runs/demonstrations etc. Challenges for Cloud Software Engineering, 2012 Slide 19
  20. 20. Problems Relatively low level model of interaction. You need to know about lots of stuff (such as certificates, security keys, etc. that have nothing to do with your work) Performance is difficult to predict License issues Costs may be high for high volumes of data transfer in/out of clouds Challenges for Cloud Software Engineering, 2012 Slide 20
  21. 21. Software engineering challenge Build a software development environment that radically simplifies hosting scientific applications on a range of clouds Challenges for Cloud Software Engineering, 2012 Slide 21
  22. 22. Cloud interfacing You build applications that make use ofthe cloud providers APIs (PaaS) to accesscommon services. ‘High-level’ cloud awareness. The platformtakes care of scaleability on demand. Limited programming language support Challenges for Cloud Software Engineering, 2012 Slide 22
  23. 23. Challenge is using existing PaaS APIs for computationally-intensive applications –  Systems are set up to support web-based applications –  Google Apps – 30 second time limit on tasks Significant application change required Several experiments here – mixed conclusions. Challenges for Cloud Software Engineering, 2012 Slide 23
  24. 24. Software engineering challenge Investigate how to adapt applications that are computation/data intensive to run within the constraints set by the PaaS interfaces from cloud providers Challenges for Cloud Software Engineering, 2012 Slide 24
  25. 25. Cloud software engineering Build ‘cloud aware’ applications which can adapt to the cloud capabilities that are available Challenges for Cloud Software Engineering, 2012 Slide 25
  26. 26. Application architecture designed for the cloud Underlying infrastructure management done in the program itself, depending on performance and cost requirements What abstractions do we need for this? Challenges for Cloud Software Engineering, 2012 Slide 26
  27. 27. Three grand challenges Challenges for Cloud Software Engineering, 2012 Slide 27
  28. 28. Programming models for the cloud Building a PaaS for high performance/ throughput computing Cloud-aware software development environments Challenges for Cloud Software Engineering, 2012 Slide 28
  29. 29. Scaling up and scaling out Traditional model to cope with increased computational demand is to scale up Supported by some cloud providers who offer different instance types. Scaling out is the more usual cloud model Challenges for Cloud Software Engineering, 2012 Slide 29
  30. 30. Coping with scaling out Redesigning algorithms and code to work with a flexible rather than a fixed number of resources –  Interaction between program and platform to know what resources are being used and what resources are available –  Structuring program into a flexible number of computational components –  Avoiding sequential computations (Amdahl’s Law) Challenges for Cloud Software Engineering, 2012 Slide 30
  31. 31. Programming models Are there programming models that can take advantage of elasticity and massive parallelism? –  Invent new models for parallel computation –  Adapt existing problems to the current map/reduce programming model Challenges for Cloud Software Engineering, 2012 Slide 31
  32. 32. Map reduce explained http://www.kchodorow.com/blog/2010/03/15/mapreduce-the-fanfiction/ Challenges for Cloud Software Engineering, 2012 Slide 32
  33. 33. Map-reduce on a slide -- Split the data into a number of computationally tractable chunks.  in = split (thedata) n = size (in) -- MAP. For each chunk, process in parallel  parfor i = 1 to n do {      out (i) = F (in (i)) } -- REDUCE. Apply a function R to the output that computes the required result -- (of course, A can be a list) A = R (out) Challenges for Cloud Software Engineering, 2012 Slide 33
  34. 34. Map-reduce assumptions Input data set can be split into chunks for independent parallel processing The reduce action is a relatively simple, fast computation Clearly, does not fit all types of problem Challenges for Cloud Software Engineering, 2012 Slide 34
  35. 35. Can scientific codes be re-engineered to take advantage of the map-reduce model? Elastic + static architecture Use the elastic cloud for data preparation/analysis (MAP) Use a cloud cluster for computation (REDUCE) Challenges for Cloud Software Engineering, 2012 Slide 35
  36. 36. Opportunistic computing Taking advantage of low-price/free resources when these are available? Challenges for Cloud Software Engineering, 2012 Slide 36
  37. 37. Fixed budget computation Compute till you run out of money Get as much computational capability as your budget will allow Use spot prices Stop and start instances Challenges for Cloud Software Engineering, 2012 Slide 37
  38. 38. Budget-driven programming Develop systems that take advantage of variable pricing in different clouds/at different times Potentially significant for computationally-intensive applications Challenges for Cloud Software Engineering, 2012 Slide 38
  39. 39. PaaS for Science Develop a set of APIS for data/compute intensive applications to support cloud interaction Long-running applications Cost/demand awareness Performance measurement/tuning Intermittent execution Challenges for Cloud Software Engineering, 2012 Slide 39
  40. 40. SDE for the cloud Creating an environment for the development of cloud-aware applications Hiding low level details of IaaS cloud interaction Testing environment Occasional computation Preliminary work – ELVIRA, St Andrews http://cloudresearch.jiscinvolve.org/wp/category/projects/ elvira/ Challenges for Cloud Software Engineering, 2012 Slide 40
  41. 41. Software as a service Challenges for Cloud Software Engineering, 2012 Slide 41
  42. 42. Effective way for a community to share services Scope for applying external computations to collected data – data mining, visualisation, etc. What are appropriate abstractions? How can large amounts of state be managed? Limiting data transfer between nodes As with all software reuse, the community rather than the project benefits. Needs central funding to pay for reusable element creation Challenges for Cloud Software Engineering, 2012 Slide 42
  43. 43. Conclusions Clouds will become the dominant computing environment in future so costs will fall significantly – major driver for developing for the cloud Offers a level of computational resource that cannot really be replicated in an institution Software engineering issues need to be addressed to make the most effective use of the cloud Challenges for Cloud Software Engineering, 2012 Slide 43

×