Augmenting Complex Problem Solving with Hybrid Compute Units


Published on

Combining software-based and human-based services is crucial for several complex problems that cannot be solved using software-based services alone. In this paper, we present novel methods for modeling and developing
hybrid compute units of software-based and human-based services. We discuss high-level programming elements for different types of software- and human-based service units and their relationships. In particular, we focus on novel pro-
gramming elements reflecting hybridity, collectiveness and adaptiveness properties, such as elasticity and social connection dependencies, and on-demand and pay-per-use economic properties, such as cost, quality and benefits, for complex problem solving. Based on these programming elements, we present programming constructs and patterns for building complex applications using hybrid ser-

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Augmenting Complex Problem Solving with Hybrid Compute Units

  1. 1. Augmenting Complex Problem Solving with Hybrid Compute Units Hong-Linh Truong, Hoa Khanh Dam, Aditya Ghose, Schahram Dustdar Distributed Systems Group, Vienna University of Technology University of Wollongong WESOA@ICSOC 2013, 2.12.2013 1
  2. 2. Outline  Motivation  Programming hybrid, collective adaptive systems in the cloud  Hybrid compute units   Fundamental programming elements and constructs Framework  Illustrating examples WESOA@ICSOC 2013, 2.12.2013 2
  3. 3. Hybridity and Diversity of Computing Models Human-based Computing Things-based computing Architecture Processing Unit Machine-based Computing S. Dustdar, H. Truong, “Virtualizing Software and Humans for Elastic Processes in Multiple Clouds – a Service Management Perspective”, in International Journal of Next Generation Computing, 2012 Today’s complex problems need Ad hoc networks Web of things hybridity and diversity-aware “collective SMP adaptive systems” (HDA-CAS) Comm. Grid WESOA@ICSOC 2013, 2.12.2013 3
  4. 4. Motivation -- Dynamic hybridity in a collective  Mixture of different types of resources working in concert in the same collective  Mixture of different roles performed in the same collective  Different functions based on different capabilities  Mixture of different quality from a single collective  Functions are performed under different quality  Resources possible offer different quality wrt their role  Mixture of cost/benefit models  A collective might perform a function with different costs/qualities WESOA@ICSOC 2013, 2.12.2013 4
  5. 5. Motivation – HAD-CAS  Challenges/difficulties in programming HDACAS in the cloud  Human-in-the-loop  Elasticity of costs, quality/benefits and resources  Unified framework for humans and software  Cross-platform with complex underlying APIs Open problem: there is a lack of suitable programming frameworks WESOA@ICSOC 2013, 2.12.2013 5
  6. 6. Our goals  Construct hybrid collective adaptive systems  Consist of service units abstracting software, people and things  With different service units roles (computation, data, management)  Be elastic in multi-cloud environments  Provide common programming features/APIs that enable different coordination/composition ways for CASs Our approach: hybrid compute units for collective adaptive systems (hCAS) WESOA@ICSOC 2013, 2.12.2013 6
  7. 7. Unified service unit model for capturing hybridity in a collective Consumption, ownership, provisioning, price, etc. Service model Elastic Service Unit Function The functional capability of the unit and interface to access the function Unit Dependency Modeling type of units (e.g., computation, data, monitor,) and their dependencies Elastic Capability Capabilities to be elastic under different requirements WESOA@ICSOC 2013, 2.12.2013 7 Software VolunteersProfessionals Thing People Resources
  8. 8. Hybrid compute units for CAS Hybrid compute unit (HCU): a set of service units includes software-based services, human-based services and things-based services that can be provisioned, deployed and utilized as a collective on-demand based on different quality, pricing and incentive models. This paper focuses on abstracting service units and relationships to provide fundamental programming elements for HCU WESOA@ICSOC 2013, 2.12.2013 8
  9. 9. Hybrid compute unit design – fundamental elements WESOA@ICSOC 2013, 2.12.2013 9
  10. 10. Hybrid compute unit design -Relationships Relationship Type HBS SBS TBS HCU Similarity Yes Yes Yes Yes Composition Yes Yes Yes Yes Data Dependency Yes Yes Yes Yes Control Dependency Yes Yes Yes Yes Location Dependency Yes Yes Yes Yes Forwarding Yes Yes No Yes Delegation Yes Yes No Yes Social Relation Yes No No Yes Elasticity Yes Yes No Yes WESOA@ICSOC 2013, 2.12.2013 10
  11. 11. Hybrid compute unit design – highlevel programming constructs  Current prototype: Fundamental elements can be implemented as classes in high-level programming languages   Service units and their properties Relationship constructs  But complex runtime is still open   Relationships require complex code libraries for different templates Complex integration with existing clouds WESOA@ICSOC 2013, 2.12.2013 11
  12. 12. Hybrid compute unit design – highlevel programming constructs Relationships Constructs Similarity similarity(U; V; criteria) ?similarity(U; x; criteria) Data Dependency datadependency(U;D; [M; ]V ) ?datadependency(x;D;M; V ), ?datadependency(U;D;M; x) ?datadependency(U;D; x[c]; V ) Location locationdependency(U; V; ctx; path) Brokering delegate(U; task; V ) forward(U; task; V ) ?delegate(U; task; x) SocialRelation socialrelation(U; V; ctx; path) ?socialrelation(U; x; distance; ctx; path). Elasticity ?elasticity(U; elasticityReq; x) WESOA@ICSOC 2013, 2.12.2013 12
  13. 13. Towards programming framework for hCAS HDA-CAS Coordination and Composition Models [ICSOC12] WESOA@ICSOC 2013, 2.12.2013 13
  14. 14. Illustrating Example  Evaluating quality of data (QoD) in simulations/data analytics processes  Typical components in simulation workflows/data analytics preprocessing solving postprocessing data analysis  Goal using hCAS to support quality of data control of the simulations/data analytics  Reasons: reducing wasting time and computational cost, dealing with tacit quality evaluation, etc. WESOA@ICSOC 2013, 2.12.2013 14
  15. 15. hCAS and simulations  Only SBS is needed,  for example, in the QoD evaluation step before pre-processing  SBS or HBS is used interchangeably or in combination  for example, in the QoD evaluation after preprocessing,  Only HBS is used  e.g., in QoD evaluation after solving. WESOA@ICSOC 2013, 2.12.2013 15
  16. 16. hCAS and simulation workflows/data analytics hCAS WESOA@ICSOC 2013, 2.12.2013 16
  17. 17. Excerpt -- Programming elasticity and collectiveness in solving steps SBS preprocessingUnit =new SBS() ; //create an instance of software unit File data =new File(fileName); //... //create another unit for QoD evaluation SBS qodEvalUnit = new SBS(); ArrayList paraType = new ArrayList(); paraType.add(File.class.getName()); //the unit must support qodEvaluate function which returns a value in [0,1] qodEvalUnit.setFunction("qodEvaluate",paraType, Double.class.getName()); //... //call qod evaluation unit Double result =(Double)qodEvalUnit.execute("qodEvaluate",params); … //call preprocessing activity if QoD is satisfied if (result > 0.9) { preProcessedData=preprocessingUnit.execute("preprocessing",params); } WESOA@ICSOC 2013, 2.12.2013 17
  18. 18. Excerpt -- Programming elasticity and collectiveness in solving steps if (qodPreProcessedData < 0.5) { //initiate a new unit ICU dataScientist = new ICU(); //create a dropbox place for sharing data DropboxAPI<WebAuthSession> scuDropbox = null; // .... DropboxAPI.DropboxLink link = scuDropbox.share("/hbscloud"); //ask the cloud of HBS to invoke the ICU VieCOMHBS vieCOMHBS = new VieCOMHBSImpl(); vieCOMHBS.startHBS(dataScientist); HBSMessage msg = new HBSMessage(); msg.setMsg("pls. use shared dropbox for communication " +link.url); vieCOMHBS.sendMessageToHBS(dataScientist, msg); } WESOA@ICSOC 2013, 2.12.2013 18
  19. 19. Excerpt – Programming elasticity and collectiveness in solving steps if (qodPreProcessedData < 0.9) { //specify some static properties of the solver SBS solverUnit2 = new SBS("solver"); solverUnit2.function.put("DIRTY_DATA", Boolean.valueOf(true)); //specify expected cost and accuracy support CostModel costModel = new CostModel(); costModel.price = 100; //max in EUR costModel.usageTime = 1000 * 60 * 60; //1 hour Quality quality = new Quality(); = Quality.ACCURACY; quality.value = 0.95; // minimum value ArrayList nfps = new ArrayList(); nfps.add(quality); nfps.add(costModel); //find solvers met quality and cost needs SBS elasticSolverUnit = (SBS)Relationship.elasticity(solverUnit2,nfps); Object solverResult2 = elasticSolverUnit.execute("solving", params1); } WESOA@ICSOC 2013, 2.12.2013 19
  20. 20. Excerpt – Programming location dependency //create a new software unit for simulation solvers SBS solverUnit = new SBS("solver"); //make sure the solver unit and the preprocessing unit are in the same data center Relationship.locationDependency(preprocessingUnit,solverUnit,"Amaz onEC2:Europe"); ArrayList params1 = new ArrayList(); params1.add(preProcessedData); //execute solver unit with input data from preprocessing unit Object solverResult= solverUnit.execute("solving",params1); WESOA@ICSOC 2013, 2.12.2013 20
  21. 21. Excerpt – Forwarding and delegating analysis request ICU dataScientist = new ICU(); //.... ICU fUnit = new ICU(); Relationship.socialrelation(dataScientist, fUnit,1,"Linkedin:DataScienceGroup/TUWien"); Relationship.forward(data, fUnit); //... SCU studentSCU = new SCU(); //.. Relationship.delegate(data, studentSCU); WESOA@ICSOC 2013, 2.12.2013 21
  22. 22. Conclusions and Future Work Hybrid compute units for implementing hybridity and diversity-aware collective adaptive systems (HDA-CAS)  Design fundamental programming elements and constructs  We are still just at the abstract programming elements/constructs  Need to work more on mapping from language construct abstractions to runtime APIs  Runtime system working with several underlying frameworks for supporting different composition and coordination techniques  WESOA@ICSOC 2013, 2.12.2013 22
  23. 23. Many thanks! WESOA@ICSOC 2013, 2.12.2013 23