vv

1. 17
CHAPTER 2
LITERATURE SURVEY
The main objective of any researcher or a developer is to design and implement
a system successfully. During the process of developing any algorithm or a protocol or
any component for that matter has to be well tested before it is deployed in to a product
or a process. This is usually achieved through some kind of testing and validation
techniques. They can be broadly classified as simulation based or experimental based.
Though there are many testing mechanisms, most of them are simulation based [48]
[49], which lacks accuracy when deployed in real environments. Few experimental
testbeds exist, but due to their huge infrastructural cost and their insufficiency to cater
new needs and requirements demanded by technologies such as IoT, Big data, there is
still prospective for developments. Thus, this chapter attempts to survey the existing
testbeds on various kinds, identify their merits and lags, which led to the motivation for
this research.
2.1 SURVEY ON TESTBEDS
Till 2009, the developed applications [28] or algorithms or protocols are mostly
tested on simulation [50] tools. Very few researchers tested their models on
experimental setups. From the year 2010, testing the results on the experimental
testbeds took off except for the year 2011 (for which the reasons are not clear). The
rationale behind was the availability of low cost hardware units, which led to reduced
infrastructural cost [51].
The proposed research work is to design, develop, and deploy an open IoT
testbed cum development utility which offers sensor data, actuators, platforms and API
as service with enhanced performance and utilization factor. Thus this research requires

2. 18
literature review at two levels. There is a need to understand the existing testbed
frameworks, their purpose, the services, challenges or issues. Secondly to investigate
the way each service is offered, their methodology, the merits, challenges or issues.
Community oriented platforms like IrisNet [52], SensorBase [53], SenseWeb
[43], Global Sensor Network (GSN) [54], and Semantic Sensor Web (SSW) [55] have
been established which allowed users to share the data from heterogeneous data
sources.
IrisNet [52] (Internet-scale Resource-Intensive Sensor Network Services) aims
at providing a sensor web which can be accessed from anywhere, anytime fulfilling the
requirement of an IoT based system. This attempt provided multitude of sensors openly
accessible to users from all walks of life.
SensorBase [53] is centralized data storage to log sensor network data in a blog
style.
SenseWeb, an Microsoft’s creation [43] provides a generic platform to share,
query and visualize sensor data.
GSN [54] offers a general-purpose infrastructure which can be programmed
based on user needs as against usual collection only model from a central repository.
SSW [55] enables interoperability and advanced analytics for situation
awareness and other advanced applications from heterogeneous sensors. But the lack
of integration and interconnection between these networks have left the most important
data isolated and have aggravated the current problem “too much data but not enough
knowledge” furthermore. Authors D. Sakumari and H. Zhang have discussed the
essential requirements in designing a basic testbed along with the overall comparison
of existing testbeds [56].
The word testbed was almost synonymous to WSN Testbed [57] and most of
the boards used are proprietary which lead to interoperability problems. Many
application based projects were deployed using these wireless sensor based
frameworks [58] [59] [60] [61]. K. Langendoen, A. Baggio, and O. Visser [62] have
shared their failures which include lack of strenuous testing, lethargy towards software

3. 19
engineering practices, insufficient API support and lessons learnt from their
experiences like design for the worst with much more valid assumptions (including
expectations for packet error rates more than 10%), more than one node failure per
week and likely more node mobility. Most of the researchpapers discussed so far, have
used Low-Rate Wireless Personal Area Networks (LoWPAN) [63] to demonstrate their
applications.
Silvia Santini and Daniel Rauch proposed Minos (Message Identification and
Online Storage) a generic tool for sensor data acquisitionand storage as part of Desthino
(Distributed Embedded Systems Online) project [64]. Though the tool is developed
using java, it is capable of interacting only with sensor nodes running TinyOS which
makes it platform dependent.
Way back in 2008 the idea of testbed was incorporated in the field of education
[65]. P. D. Godoy, R. Cayssials and C. G. Garino have highlighted the importance of
simulated and real experimental testbeds. A WSN extended IoT based testbed have
also been developed to support teaching [66].
Sensor Data Service defaults in Windows 10 delivers data from a variety of
sensors [67] , but this service has to be started manually by the users but is based on a
proprietary operating system Windows 10.
KanseiGenei testbed exists since 2007 and still available for public use [68]. It
uses a 3-tier architecture comprising of wireless sensor nodes wired to a gateway.
M. Zorzi, A. Gluhak, S. Lange, A. Bassi [69] have attempted to provide an
architectural framework to overcome the current fragmentation and limitation of
solutions, where many “Intranets” of Things exist, towards a true “Internet” of Things,
where all devices will be part of a globally integrated system. The authors have realized
it early, the need for a unified approach for IoT when many of them were still busy
developing single applications in silos. They have admitted the agony of only small
group of enthusiasts from academia, industry and public institutions working to bring
up a unified model for IoT. They have emphasized the dire need for standardization and
ETSI have made some notable contributions.

4. 20
Few researchers worked on realizing things as smart objects, there by achieving
intelligence in the system by interconnecting them [70]. This idea helps IoT systems
achieve their real purpose of smartness [71].
The facilities needed for an experimental testbed is discussed and the need for
experimental real time testbeds [37] is well established. SenseLab- a very large scale
open Wireless Sensor Network Testbed [72] is a generic testbed which allows
experimental research of protocols used for communication and algorithms at
application level.
WISEBED was initially started in 2010 [73], later evolved to include concepts
of virtualization to the existing testbed architecture. It uses a generic XML-based
language (WiseML) to describe about the experimental setting up and for result
storage.
A federated platform for mobile data centric service development and sharing
has been developed as an attempt to address interoperability through virtualization[46].
This model requires the developer or user to write a script which is capable of parsing
the received data to suit his/her needs.
M. Nati, A. Gluhak, H. Abangar, W. Headley claims that many of the existing
IoT technologies are still tested in lab based testbeds [74] and emphasizes the need for
realism [75] in external environment and the importance of real end users in the loop.
They have created an infrastructure to support smart building based experimentation
with using TelosB motes.
Control Cube is a work presented at IPSO challenge, it is a device (bothsoftware
and hardware) that enables every-day, conventional appliances and automations to join
the IoT vision using IPv6, CoAP and the IPSO Application Framework [76]. It
highlights the mammoth task involved in developing new type of resources to support
varied devices and the CoAP implementation of TinyOS has to be greatly extended.
Moreover, severe hardware limitations demands work on careful resource optimization
(especially memory constraints) when developing firmware.

5. 21
In order to ensure, reliability of wireless communication in IoT based systems
which is spread across distributed locations, enhanced lifetime through safe and proper
utilization of energy and judicious memory utilization of resource constrained devices,
A. S. Tonneau, N. Mitton, J. Vandaele have studied the existing experimental setups,
various hardware’s they support and the flavor of services they offer [77]. They have
attempted to define the requirements of an IoT experimental testbed based on survey
of existing facilities, their purpose and functionality they offer. Researchers have also
attempted survey based on relevant technologies, protocols and applications for IoT
[78] [13]. L. Belli has attempted to bring in web interface to access experimental
testbeds with the idea of web of things with the underlying backbone being the internet
[79]. D. Guinard, V. Trifa and E. Wilde have proposed service oriented architecture for
web of things [80].
FIT IoT-LAB provides a very large-scale infrastructure suitable for testing
small wireless sensor devices and heterogeneous communicating objects. The project
is an extension of SenseLab testbed set up [81]. FIT is Future Internet of Testbeds was
the abbreviation. The title was changed to Future IoT in 2017 and recently they have
changed it to Future Internet Testing facility. Research has been carried out with FIT
IoT-LAB for various experimentation needs [82] [39].
The possible benefits of collaboration between simulation tools and real
physical testbeds, their trade-offs and the role of simulation tools in bridging scalability
issues is highlighted in the paper [83].
A new application genre for IoT, IoT-based E-Learning testbed [84] capable of
stimulating the learner's motivation by using implemented smart box, microwave
sensor module and decision tree is discussed.
Most of the existing experimental testbeds are extrapolation of wireless sensor
networks [1] which are made of proprietary motes leading to interoperability issues
claims L. Mainetti, L. Patrono, and A. Vilei. The authors G. Z. Papadopoulos, J.
Beaudaux, A. Gallais, T. Noel and G. Schreiner have emphasized the importance of
experimental testbeds compared to simulation models but the setup used comprises of
motes leading to vendor lock in and difficulty in migration or scaling [9].

6. 22
There are some commercially available services such as ThingSpeak, Ubidots,
etc., that provide a way to send the data from IoT systems to the cloud. The authors A.
I. Abdul-Rahman, C. A. Graves have exploited the existing ThingSpeak Cloud and
Service science to achieve reusability by sharing data collected and stored [85]. Earlier
these services used the HTTP application layer protocol and have started to support
MQTT recently. However, these services still require the application generating the
data to be developed by the user and then the data will be available to use over the
internet. This provides a way of ensuring that the data is available from anywhere but
does not reduce the burden onthe analyst of having to develop or implement the system
that generates the data.
There are publicly available channels in ThingSpeak [86], used by other
application developers but these channels do not categorize the data on the basis of
sensors. Those channels are also not maintained by the service provider to ensure
constant availability of data in them.
Moreover, ThingSpeak allows only one update every 15 seconds, when
uploading data to an API [87], but there are few applications which need more frequent
updates or on a specific change in the event in other words data.
The author have described that IoT will be the driving force for data as a service
[45]. In line withprevious discussions, IoT is all about communicationand cooperation
between objects to implement the task at hand. A. Jimenez-Gonzalez, J.R.M.D. Dios,
A. Ollero have explored the possibilities of cooperative perception of sensors in an
integrated Testbed [88]. J. R. Martinez-de Dios, A. Jimenez-Gonzalez, A. De San
Bernabe, A. Ollero emphasize the importance of integration of heterogeneous objects
and technologies to achieve modularity and ease of access of testbeds [89]. All these
frameworks offer sensor data as service in wide variety of ways, but the point most of
the approaches missed was IoT perspective of the data, standardized data formats to
enhance reusability, optimized methods of data collection [90], storage and measures
on erroneous data.
G. Z. Papadopoulos, A. Gallais, G. Schreiner, E. Jou have reviewed 674 papers,
out of which 596 papers belong to IoT, WSN and Ad hoc networks [38]. 561 papers
have provided theoretical results for their study under consideration. 284 papers out of

7. 23
596 have embraced simulation and 392 papers on experimental testing. But among the
392 papers most of them belong to the category of WSN or Ad hoc. Even if the paper
was related to IoT, it was tested on an extrapolated WSN testbed, but not on an
exclusive IoT testbed [91].
Though the testing on experimental testbed has considerably risen, but there are
challenges which need addressing. It requires frequent flashing, reboot after flash, and
reload the firmware, debug, sooner or later to retrieve measurements. The most
importantly requires undeniable dedicated skill set and prior knowledge of the set up.
Setting up is equally difficult and time consuming. Also mobilize human resources
other than researchers is equivalently tough. [88]
In order to overcome these challenges, there is inclination towards simulations
[49] based testing grounds. But these advantages come with a cost. Simulations [92]
[93] usually hide the limitations or constraints imposed by actual components in the
name of abstraction. For example, provide unlimited memory, powerful computation
resource. Having said that, the probability of node failure and energy drains are never
taken in to consideration. This makes the realism of test results questionable. Also it
complicates by underestimating the complexity of their contribution, leading to false
promising results.
Some of the experimental frameworks provided on-demand resource sharing
[94] and facility for developing value added services but the issue of proprietorship and
interoperability was not addressed in depth.
IETF have come up with internet draft that describes about importance of
resource discovery and its benefits. This draft emphasizes the importance of resource
sharing and the benefits of dynamic discovery. It is in its infancy in the field of IoT.
The underlying protocol to identify the resources can be chosen from available pool.
The most deployed protocols are CoAP [95] and MQTT among others.
P. Fremantle has provided architecture [96] level detailing of IoT. J. H. Huh,
D. H. Kim and K. Jong-Deok have proposed a similar such model named as NEWS, N:
Neutrality, E: Ease, W:Web, S:Shareable and [97] have provided a top level discussion
and they are yet to evolve the experimental results.

8. 24
M. Chernyshev, Z. Baig, O. Bello, S. Zeadally in 2018 have provided a
comparative analysis and open research challenges of existing IoT based simulation
tools [48] grouped in the capacity of exposure of the IoT architecture layers.
Of late IoT related events [98], consortium having been formulated to discuss
the issues and formulate solutions keeping in mind the need for standardization.
2.2 COMPREHENSIVE COMPARSION OF PROPOSED VS. EXISTING
TESTBEDS
Thus this research work is an attempt to address all the above mentioned issues
and there by achieve an open source heterogeneous testbed utility considering
interoperability [99] among open source devices. Figure 2.1 provides the
comprehensive comparison of the proposed system against existing testbed
frameworks.
Figure 2.1 Comparison chart of proposed vs. existing testbeds

9. 25
Thus the list of gaps or research challenges to address is:
 Sensors and Actuators are mere input and output not as service/utility
 Vendor lock-in- Proprietary
 Fear of obsolescence
 Shifting an issue
 Deters innovation and dynamicity
 Lack of interoperability
 Lack of standardization
 Lack of open source entities
 Insufficient control
 Lack of conflict resolve/concurrency (Actuators)
 Obtaining the data will be useless, if the data is in a proprietary format
 Simulation based environments
 WSN extrapolated frameworks
This research work also exploits the benefits of publish-subscribe model where
each device subscribes to central broker if it is interested in a specific topic without
worrying about details of its co-existing nodes. The research work highlights
improvised methodology, not only to achieve sensor data as service, but also actuator
as service, platform as service and API as service. The optimized ways of achieving the
same is also dealt. The proposed research work also gives insight of how the message
is packed, flexibility in intervals of data collection and mode, error handling and
providing appropriate visualization for each sensor data. Moreover, the proposed
testbed framework have reduced the learning curve the data analysts have to undergo,
provided categorization based on sensors granulated at parameter level, better
visualization, and had removed the dependency of any coding skills or hardware
knowledge required to create the application that generates the data. The ranking and
testbed utilization model is also developed to prove the reliability and performance of
the proposed framework.