Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

In stream Big Data processing

1,166 views

Published on

Cloud platforms integrate three pillars: messaging, coordination of workers and data. This research investigates whether functional programming languages (FPLs) have any special merit when it comes to the implementation of cloud computing platforms. We have implemented an optimistic message queue (CMQ) and an in memory tuple space (cwmwl) for the coordination of workers that can be used as artefact to proof or disproof the special merit of FPLs in computing clouds. We now address the problem of in-stream processing of Big Data (for example social network streams, logistic data or data mashups) to investigate the integration with Big Data.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

In stream Big Data processing

  1. 1. In stream Big Data processing. Joerg Fritsch, J.Fritsch@cs.cardiff.ac.uk School of Computer Science and Informatics, Cardiff University UK Introduction Cloud platforms integrate three pillars: messaging, coordination of workers and data. This research investigates whether functional programming languages (FPLs) have any special merit when it comes to the implementation of cloud computing platforms. We have implemented an optimistic message queue (CMQ) and an in memory tuple space (cwmwl) for the coordination of workers that can be used as artefact to proof or disproof the special merit of FPLs in computing clouds. We now address the problem of in-stream processing of Big Data (for example social network streams, logistic data or data mashups) to investigate the integration with Big Data. Hypothesis Functional Programming Languages (FPLs), with their inherent parallelism and easy-access, lightweight units of scale, working together with a LINDA coordination model, provide an ideal basis for a cloud computing platform (PaaS) consisting of messaging, coordination and the in-memory integration with data (flows). Optimistic messaging, coordination and data. Figure 1 applies the principles learned from FPLs (e.g. shared nothing, modularity, no state) to create a framework for in transit processing of big data. Figure 1 Logical Diagram showing combination of optimistic messaging and the in memory tuple space for in stream data processing. The functional programming language Lua is used for data processing such as data masking or “scrubbing”. Implications of our work IaaS, PaaS and SaaS are today’s means to create computing clouds, --the illusion of unlimited elastic computing resources. Map Reduce (MapR) are today’s answer to the processing of Big Data. However, both worlds ( _aaS and MapR) are sold as separate offerings because they do not integrate with each other and none of the solutions are truly elastic. Our work holds special significance in the development of true elasticity by minimizing the units of scale and in the integration of the provisioned units of scale with real time (big) data processing. Papers Fritsch J. Walker C. (2012), “CMQ - A lightweight, asynchronous high-performance messaging queue for the cloud”, Journal of Cloud Computing: Advances, Systems and Applications, 1(1) Fritsch J. Walker C. (2013), “Cwmwl, a LINDA-based PaaS fabric for the cloud”, Journal of Communications, SI on Cloud and Big Data (to be published)

×