TUW- 184.742 Emerging Dynamic Distributed Systems and Challenges for Advanced Services Engineering

Uploaded on

This presentation is part of the course "184.742 Advanced Services Engineering" at The Vienna University of Technology, in Winter Semester 2012. Check the course at: …

This presentation is part of the course "184.742 Advanced Services Engineering" at The Vienna University of Technology, in Winter Semester 2012. Check the course at: http://www.infosys.tuwien.ac.at/teaching/courses/ase/

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Advanced Services Engineering, WS 2012 Emerging Dynamic Distributed Systems and Challenges for Advanced Services Engineering Hong-Linh Truong Distributed Systems Group, Vienna University of Technology truong@dsg.tuwien.ac.at http://www.infosys.tuwien.ac.at/staff/truongASE WS 2012 1
  • 2. Outline Today‘s Internet Computing Some emerging models – properties and issues  Data provisioning models  Computational infrastructures/frameworks provisioning  Human computation provisioning Internet-scale service engineering Single service/platform engineeringASE WS 2012 2
  • 3. Today‘s Internet Computing  Internet infrastructure and software connect contents, things, and people, each has different roles (computation, sensing, analytics, etc.) Size does Any * access Economic matter behaviour does matter factors do matter Things Large-scale Unpredictable On-demand, pay- interactions workload as-you-go Big data Scalability ComplexSoftware People generated contract Internet infrastructure and Big quantities to be managed software Hard to control quality ASE WS 2012 3
  • 4. Today‘s Internet Computing Social Technologies and computing computing models Cloud Service Computing Computing Peer-to- Distributed Peer Computing Computing converge Things Emerging forms of computing introduces models, systems Software People and applicationsASE WS 2012 4
  • 6. Some emerging data provisioning models (1) Large (near- • Satellites and environmental/city sensor networks (e.g., from specific orgs/countries) ) realtime • Machine-to-machine (e.g., from companies) • Social media (e.g., from people + platform providers) data • Open science and engineering data sets Open data • Open government data Marketable • Statistics and business data • Commercial data in general data Data are assetsASE WS 2012 6
  • 7. Some emerging data provisioning models (2) A lot A few A lot Things Social Data/Service Platforms Platforms Data Profiling Data and Enrichment Storage APPs Environtments ... Data Data Analytics Query Infrastructures ....ASE WS 2012 7
  • 8. Examples of large-scale (near-) realtime dataASE WS 2012 8
  • 9. Large-scale (near-)realtime data: properties and issuesSome properties Some issues Having massive data  Timely analytics Requiring large-scale, big  Performance and (near-) real time scalability processing and storing  Quality control capabilities  Handle of unknown data Enabling predictive and patterns realtime data analytics  Benefit/cost versus quality tradeoffsASE WS 2012 9
  • 10. Example of open dataASE WS 2012 10
  • 11. Open data: properties and issuesSome properties Some issues Having large, multiple  Fine-grained content data sources but mainly search static data  Balance between Having good quality processing cost and control in many cases performance Usually providing the data as a whole setASE WS 2012 11
  • 12. Marketable data examplesASE WS 2012 12
  • 13. Marketable data: properties and issuesSome properties Some issues Can be large, multiple  Multiple levels of data sources but mainly service/data contracts static data  Compatible with other Having good quality data sources w.r.t. control contract Have strong data contract  Cost w.r.t. up-to-date terms data Some do not offer the whole datasetASE WS 2012 13
  • 14. Emerging computational infrastructure/platform provisioning models Infrastructure-as-a-Service  Machine-as-a service  Storage as a Service  Database as a Service Platform-as-a-Service  Middleware  Computational frameworks Software Defined NetworkingASE WS 2012 14
  • 15. Examples of Infrastructure-as-a- ServiceSource: Hong Linh Truong, Schahram Dustdar: Cloud computing for small research groups in computational science and engineering: current status and outlook. Computing 91(1): 75-91 (2011) Amazon S3 Microsoft AruzeAnd more MongoLab OKEANOS ASE WS 2012 15
  • 16. Examples of Platform-as-a-ServiceSource: Hong Linh Truong, Schahram Dustdar: Cloud computing for small research groups in computational science and engineering: current status and outlook. Computing 91(1): 75-91 (2011) Amazon Elastic MapReduceAnd more StormMQ Globus Online (GO) ASE WS 2012 16
  • 17. Examples of multiple cloudsSource: Katarzyna Keahey, Mauricio Tsugawa, Andrea Matsunaga, and Jose Fortes. 2009. Sky Computing. IEEE InternetComputing 13, 5 (September 2009), 43-51. DOI=10.1109/MIC.2009.94 http://dx.doi.org/10.1109/MIC.2009.94  aaa SOCloud WS 2011 17
  • 18. Emerging computational infrastructure/platform provisioning models– properties and issuesSome properties Some issues Rich types of services  On-demand information from multiple providers management from  Better choices in terms of multiple sources functions and costs  APIs complexity Concepts are similar but  Cross-vendor integration diverse APIs  Data locality Strong dependencies/tight ecosystemsASE WS 2012 18
  • 19. Emerging human computation models Crowdsourcing platforms  (Anonymous) people computing capabilities exploited via task bids Individual Compute Unit  An individual is treated like „a processor“ or “functional unit“. A service can wrap human capabilities to support the communication and coordination of tasks Social Compute Unit  A set of people and software that are initiated and provisioned as a service for solving tasksThe main point: humans are a computing elementASE WS 2012 19
  • 20. Examples of human computation (1) Source: Salman Ahmad, Alexis Battle, Zahan Malkani, Sepandar D. Kamvar: The jabberwocky programming environment for structured social computing. UIST 2011: 53-64ASE WS 2012 20
  • 21. Examples of human computation (2) Source: Daniel W. Barowy, Charlie Curtsinger, Emery D. Berger, Andrew McGregor: AutoMan: a platform for integrating human-based and digital computation. OOPSLA 2012: 639-654ASE WS 2012 21
  • 22. Examples of human computation (3) Source: Muhammad Z.C. Candra, Rostyslav Zabolotnyi, Hong-Linh Truong, and Schahram Dustdar, Virtualizing Software and Human for Elastic Hybrid Services, Web Services Handbook, (c)Springer-Verlag, 2012.ASE WS 2012 22
  • 23. Human computation models – properties and issuesSome properties Some issues Huge number of people  Quality control Capabilities might not  Reliability assurance know in advance  Proactive, on-demand Simple coordination acquisition models  Incentive strategiesASE WS 2012 23
  • 24. Summary of emerging models wrt advanced service-based systems Engineering advanced service- based systems utilize/consist of Emerging data provisioning models Things Emerging data provisioning models Software PeopleEmerging computational Emerging human Emerging data infrastructure/platform computation provisioning provisioning models models modelsASE WS 2012 24
  • 26. Recall our motivating example (1) Infrastructure/Internet of Things Internet/public cloud Organization-specific boundary boundary Equipment Operation and Maintenance Emergency Management Near Enterprise Civil protection realtime analytics Resource Planning Building Operation Predictive data Optimization analytics Tracking/Log istics Visual Analytics Infrastructure Monitoring ...Cities, e.g. including:10000+ buildings1000000+ sensors Can we combine open government data with building monitoring data? ASE WS 2012 26
  • 27. Recall our motivating Can we combine them with open government example (2) data?Soilmoistureanalysis forSentinel-1A lot of input data (L0):~2.7 TB per dayA lot of results (L1, L2): e.g., L1 has ~140 MB per day for a grid of 1kmx1km Michael Hornacek,Wolfgang Wagner, Daniel Sabel, Hong-Linh Truong, Paul Snoeij, Thomas Hahmann, Erhard Diedrich, Marcela Doubkova, Potential for High Resolution Systematic Global Surface Soil Moisture Retrieval Via Change Detection Using Sentinel-1, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, April, 2012 ASE WS 2012 27
  • 28. Recall our motivating example (3) Source: http://www.undata-api.org/ Source: http://www.strikeiron.com/Catalog/StrikeIronServices.aspx Source: http://docs.gnip.com/w/page/23722723/Introduction- to-GnipASE WS 2012 28
  • 29. Discussion time: WHICH OPPORTUNITIES DO YOU SEE?ASE WS 2012 29
  • 30. Internet-scale service engineering - - the elasticityASE WS 2012 30
  • 31. Internet-scale service engineering - - big/near-real time data impact Which are data concerns that impact the data processing? How to use data concerns to optimize data analytics and service provisioning? How to use available data assets for advanced services in an elastic manner? What are the role of human-based servies in dealing with complex data analytics?ASE WS 2012 31
  • 32. Internet-scale service engineering - - Steps Single service/platform engineeringService units for representing Provisioning of fundamental Engineering with single fundamental things, people service units service units and software Understanding Properties/Concerns Data /Service/Application Monitoring, evaluation and Utilization of data/serviceconcerns; their dependencies provisioning of concerns concerns Large-scale, multi-platform services engineering Identify design units, selection development and Identify the scale,platform/application of existing service Integration, complexity and *city problems units; Optimization ASE WS 2012 32
  • 33. Discussion time: WHAT ARE MISSING?ASE WS 2012 33
  • 34. Single service/platform engineering – service unit (1)  The service model and the unit concept can be applied to things, people and software Consumption, ownership, Service provisioning, price, etc. model Service unit „basic component“/“basic function“ modeling Unit Concept and descriptionASE WS 2012 34
  • 35. Single service/platform engineering – service units (2) Source: Stefan Tai, Philipp Leitner, Schahram Dustdar: Design by Units: Abstractions for Human and Compute Resources for Elastic Systems. IEEE Internet Computing 16(4): 84-88 (2012)ASE WS 2012 35
  • 36. Single service/platform engineering – service unit provisioning  Provisioning software under services  Provisioning things under services  Provisioning human under services  Crowd platforms of massive numbers of individuals  Individual Compute Unit (ICU)  Social Compute Unit (SCU)1. Mark Turner, David Budgen, and Pearl Brereton. 2003. Turning Software into a Service. Computer 36, 10 (October 2003), 38-44. DOI=10.1109/MC.2003.1236470 http://dx.doi.org/10.1109/MC.2003.12364702. Luigi Atzori, Antonio Iera, and Giacomo Morabito. 2010. The Internet of Things: A survey. Comput. Netw. 54, 15 (October 2010), 2787-2805. DOI=10.1016/j.comnet.2010.05.010 http://dx.doi.org/10.1016/j.comnet.2010.05.0103. Dominique Guinard, Vlad Trifa, Stamatis Karnouskos, Patrik Spiess, Domnic Savio: Interacting with the SOA-Based Internet of Things: Discovery, Query, Selection, and On-Demand Provisioning of Web Services. IEEE T. Services Computing 3(3): 223-235 (2010)4. Schahram Dustdar, Kamal Bhattacharya: The Social Compute Unit. IEEE Internet Computing 15(3): 64-69 (2011)5. Hong-Linh Truong, Schahram Dustdar, Kamal Bhattacharya "Programming Hybrid Services in the Cloud", Springer-Verlag, 10th International Conference on Service-oriented Computing (ICSOC 2012), November 12-16, 2012, Shanghai, China ASE WS 2012 36
  • 37. Single service/platform engineering – examples (1) Service engineering with a single system/platform  Using Excel to access Azure datamarket places  Using Boto to access data in Amazon S3  Using Hadoop within a cluster to process local data  Using workflows to process data (e.g., Trident/Taverna/ASKALON)  Using StormMQ to store messagesASE WS 2012 37
  • 38. Single service/platform engineering – examples (2)ASE WS 2012 38
  • 39. Internet-scale multi-platform services engineering – required technologies Middleware (e.g., Workflows (e.g., StormMQ) Trident) Crowd platforms, Data services (e.g., human-based service Azure, S3) platforms(e.g., Mturks, VieCOM) Internet-scale, Data Multi-platform Services Billing/Monitoring analysis/Computation (e.g., services in cluster Engineering for thecurrencycloud) (e.g., Hadoop) Software, Things and PeopleASE WS 2012 39
  • 40. Discussion time WHAT ARE MISSING?ASE WS 2012 40
  • 41. Exercises Read papers mentioned in slides  Get their main ideas Check services mentioned in examples  Examine capabilities of the mentioned services  Including price models and underlying technologies  Examine their size and scale  Examine their ecosystems and dependencies Work on possible categories of single service units that are useful for your work  Some common service units with capabilities and providersASE WS 2012 41
  • 42. Thanks for your attention Hong-Linh Truong Distributed Systems Group Vienna University of Technology truong@dsg.tuwien.ac.at http://www.infosys.tuwien.ac.at/staff/truongASE WS 2012 42