Building Blocks of Mayan: Componentizing the eScience Workflows Through Software-Defined Service Composition

308 views

Published on

eScience consists of computation-intensive workflows executing on highly distributed networks. Service compositions aggregate web services to automate scientific and enterprise business processes. Along with the increased demand for data quality and Quality of Service (QoS) for an accurate outcome in a shorter completion time, execution of the eScience workflows and service compositions are also required to be distributed efficiently across various geo-distributed nodes. This paper presents Mayan, a Software-Defined Networking (SDN) based approach for service composition.

Mayan i) facilitates an adaptive execution of scientific workflows, ii) offers a more efficient service composition by leveraging distributed execution frameworks, in addition to the traditional web service engines, and iii) enables a very large-scale reliable service composition by finding and consuming the current best-fit among the multiple implementations or deployments of the same service.

Published in: Software
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
308
On SlideShare
0
From Embeds
0
Number of Embeds
237
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Building Blocks of Mayan: Componentizing the eScience Workflows Through Software-Defined Service Composition

  1. 1. Building Blocks of Mayan: Componentizing the eScience Workflows Through Software-Defined Service Composition Pradeeban Kathiravelu*, Tihana Galinac Grbac+, Luís Veiga* *INESC-ID Lisboa & Instituto Superior Técnico, Universidade de Lisboa, Portugal +University of Rijeka, Croatia 23rd IEEE International Conference on Web Services (ICWS 2016) June 27 - July 2, 2016, San Francisco, USA. Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 1 / 28
  2. 2. Overview 1 Introduction 2 Mayan Approach 3 Evaluation 4 Conclusion Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 2 / 28
  3. 3. Introduction Introduction eScience workflows Computation-intensive. Execute on highly distributed networks. Complex service compositions aggregating web services To automate scientific and enterprise business processes. Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 3 / 28
  4. 4. Introduction Motivation Increasing demand for Data quality and Quality of Service (QoS). Better Performance (Shorter completion times and higher throughput). Geo-distribution (workflows and compositions). Need for additional control and flexibility. Exploring Trade-off: Efficiency vs. Accuracy. Leveraging Software-Defined Approaches (from SDN). Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 4 / 28
  5. 5. Introduction Goals Scalable Distributed Executions. High Scalability. Better orchestration. Data Quality Assurance. Multi-Tenanted Environments. Isolation Guarantees. Differentiated Quality of Service (QoS). Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 5 / 28
  6. 6. Introduction Contributions Support for, Adaptive execution of scientific workflows. Flexible service composition. Reliable large-scale service composition. Efficient selection of service instances. Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 6 / 28
  7. 7. Mayan Approach Mayan Extensible SDN approach for cloud-scale service composition Driven by: Loose coupling Message-oriented Middleware (MOM) Availability of a logically centralized control plane Leveraging OpenDaylight SDN controller as the core. Modular, as OSGi bundles. Additional advanced features. State of executions and transactions stored in the controller distributed data tree. Clustered and federated deployments. Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 7 / 28
  8. 8. Mayan Approach Services as the building blocks of Mayan Prototypical Example: Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 8 / 28
  9. 9. Mayan Approach Software-Defined Service Composition Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 9 / 28
  10. 10. Mayan Approach Multiple Implementations and Deployments of a Service Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 10 / 28
  11. 11. Mayan Approach Software-Defined Service Composition Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 11 / 28
  12. 12. Mayan Approach Services as the building blocks of Mayan Prototypical Example: Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 12 / 28
  13. 13. Mayan Approach Too many requests on the fly? Prototypical Example: Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 13 / 28
  14. 14. Mayan Approach Alternative Deployment/Implementation Prototypical Example: Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 14 / 28
  15. 15. Mayan Approach Mayan Services Registry: Modelling Language Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 15 / 28
  16. 16. Mayan Approach Service Composition Representation <Service3,(<Service1, Input1>, <Service2, Input2>)> Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 16 / 28
  17. 17. Mayan Approach Alternative Implementations and Deployments Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 17 / 28
  18. 18. Mayan Approach Mayan Higher Level Deployment Architecture: Multi-Domain Workflows Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 18 / 28
  19. 19. Mayan Approach Connecting Services View with the Network View Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 19 / 28
  20. 20. Mayan Approach Connecting Services View with the Network View Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 20 / 28
  21. 21. Evaluation Evaluation System Configurations Evaluation Approach: Smaller physical deployments in a cluster. Larger deployments as simulations and emulations (Mininet). Evaluated Deployment: Service Composition Implementations. Web services frameworks. Apache Hadoop MapReduce. Hazelcast In-Memory Data Grid. OpenDaylight SDN Controller. Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 21 / 28
  22. 22. Evaluation Preliminary Assessments A workflow performing distributed data cleaning and consolidation [PK 2015]. A distributed web service composition. vs. Mayan approach with the extended SDN architecture. Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 22 / 28
  23. 23. Evaluation Speedup and Horizontal Scalability No negative scalability in larger distributions. 100% more positive scalability for larger deployments. Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 23 / 28
  24. 24. Evaluation Memory consumption in the Service Nodes Initial coordination overhead in memory for smaller deployments. Minimal overhead for larger deployments. Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 24 / 28
  25. 25. Conclusion Related Work MapReduce for efficient service compositions [SD 2014]. But we should not forget the registry! Palantir: SDN for MapReduce performance with the network proximity data [ZY 2014]. A multi-domain deployment of SDN for community networks [PK 2016]. Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 25 / 28
  26. 26. Conclusion Conclusion SDN-based approach that enables large scale flexibility with performance Components in eScience workflows as building blocks of a distributed platform. Service composition with web services and distributed execution frameworks. Multi-tenanted multi-domain executions. Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 26 / 28
  27. 27. Conclusion Conclusion SDN-based approach that enables large scale flexibility with performance Components in eScience workflows as building blocks of a distributed platform. Service composition with web services and distributed execution frameworks. Multi-tenanted multi-domain executions. Future Work Mayan should further be deployed and evaluated on physical geo-distributed nodes. Extending Software-defined service composition for the network functions in service composition of middlebox actions. Load balancing. Firewalls. Adapting as an NFV framework for service function chaining. Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 27 / 28
  28. 28. Conclusion References PK 2015 Kathiravelu, Pradeeban, Helena Galhardas, and Luís Veiga. "∂u∂u Multi-Tenanted Framework: Distributed Near Duplicate Detection for Big Data." On the Move to Meaningful Internet Systems: OTM 2015 Conferences. Springer International Publishing, 2015. SD 2014 Deng, Shuiguang, et al. "Top-Automatic Service Composition: A Parallel Method for Large-Scale Service Sets." Automation Science and Engineering, IEEE Transactions on 11.3 (2014): 891-905. ZY 2014 Yu, Ze, et al. "Palantir: Reseizing network proximity in large-scale distributed computing frameworks using sdn." 2014 IEEE 7th International Conference on Cloud Computing (CLOUD). IEEE, 2014. PK 2016 Kathiravelu, Pradeeban, and Luıs Veiga. "CHIEF: Controller Farm for Clouds of Software-Defined Community Networks." Software Defined Systems (SDS), 2016 IEEE International Symposium on. IEEE, 2016. Thank you! Questions? Pradeeban Kathiravelu (IST-ULisboa) Software-Defined Service Composition 28 / 28

×