Opportunities and Challenges of Large-scale IoT Data Analytics
Opportunities and Challenges of
Large-scale IoT Data Analytics
Institute for Communication Systems (ICS)/
5G Innovation Centre
University of Surrey
Guildford, United Kingdom
ASEAN IoT Innovation Forum, Kuala Lumpur,
Malaysia, August 2015
2P. Barnaghi et al., "Digital Technology Adoption in the Smart Built Environment", IET Sector Technical Briefing, The Institution of Engineering and Technology
(IET), I. Borthwick (editor), March 2015.
Internet of Things: The story so far
Wireless Sensor and
, solutions for
efficiency, routing, …
vertical applications, early
concepts and demos, …
More products, more
solutions for control and
Future: Cloud, Big (IoT) Data
Analytics, Interoperability, Enhanced
Cellular/Wireless Com. for IoT,
Real-world operational use-cases
and Industry and B2B
P. Barnaghi, A. Sheth, "Internet of Things: the story so far", IEEE IoT Newsletter, September 2014.
“Each single data item is important.”
“Relying merely on data from sources that are
unevenly distributed, without considering
background information or social context, can
lead to imbalanced interpretations and
− Multi-modal and heterogeneous
− Noisy and incomplete
− Time and location dependent
− Dynamic and varies in quality
− Crowed sourced data can be unreliable
− Requires (near-) real-time analysis
− Privacy and security are important issues
− Data can be biased- we need to know our data!
Source: The IET Technical Report, Digital Technology Adoption in the Smart Built Environment: Challenges and opportunities of
data driven systems for building, community and city-scale applications,
“The ultimate goal is transforming the raw data
to insights and actionable knowledge and/or
creating effective representation forms for
machines and also human users and creating
This usually requires data from multiple sources,
(near-) real time analytics and visualisation
and/or semantic representations.
“Data will come from various source and from
different platforms and various systems.”
This requires an ecosystem of IoT systems with
several backend support components (e.g.
pub/sub, storage, discovery, and access services).
Semantic interoperability is also a key
The slide adapted from the IoT talk given by Jan Holler of Ericsson at IoT Week 2015 in Lisbon.
Search on the Internet/Web in the early days
Accessing IoT data
“ The internet/web norm (for now) is often to use
an interface to search for the data; the search
engines are usually information locators – return
the link to the information; IoT data access is
more opportunistic and context aware”.
The IoT requires context-aware and opportunistic
push mechanism, dynamic device/resource
associations and (software-defined) data routing
IoT environments are usually dynamic and (near-) real-
Off-line Data analytics
Data analytics in dynamic environments
Image sources: ABC Australia and 2dolphins.com
What type of problems we expect to solve
using the IoT and data analytics solutions?
14Source LAT Times, http://documents.latimes.com/la-2013/
A smart City example
Future cities: A view from 1998
Back to the Future: 2013
Source: thestar.com.my & skyscrappercity.com
Applications and potentials
− Analysis of thousands of traffic, pollution, weather, congestion,
public transport, waste and event sensory data to provide
better transport and city management.
− Converting smart meter readings to information that can help
prediction and balance of power consumption in a city.
− Monitoring elderly homes, personal and public healthcare
− Event and incident analysis and prediction using (near) real-
time data collected by citizen and device sensors.
− Turning social media data (e.g.Tweets) related to city issues
into event and sentiment analysis.
− Any many more…
City event extraction from social streams
Tweets from a city
AggregationOSM LocationsOSM Locations SCRIBE ontologySCRIBE ontology
511.org hierarchy511.org hierarchy
City Event ExtractionCity Event Annotation
P. Anantharam, P. Barnaghi, K. Thirunarayan, A.P. Sheth, "Extracting City Traffic Events from Social Streams", ACM Trans. on Intelligent
Systems and Technology, 2015.
Collaboration with Kno.e.sis, Wright State University
Hierarchical spatial structure of geohash for
representing locations with variable precision.
Here the location string is 5H34
0 1 2 3 4 5 6
7 8 9 B C D E
F G H I J K L
2 3 4
5 6 8 9
0 1 2 3 4
5 6 7
0 1 2
3 4 5
6 7 8
Social media analysis
Tweets from a city
P. Anantharam, P. Barnaghi, K. Thirunarayan, A. Sheth, "Extracting city events from social streams,“, ACM Transactions on TICS, 2014.
Social media analysis (deep learning –
Accumulated and connected knowledge?
Image courtesy: IEEE Spectrum
Users in control or losing control?
Image source: Julian Walker, Flicker
Data Analytics solutions for IoT data
− Great opportunities and many applications;
− Enhanced and (near-) real-time insights;
− Supporting more automated decision making and in-depth
analysis of events and occurrences by combining various
sources of data;
− Providing more and better information to citizens;
− We need to know our data and its context (density, quality,
− Open Data (there needs to be more real-time data)
− Complementary data
− Citizens in control
− Transparency and data management issues (privacy, security,
− Reliability and dependability of the systems
− IoT data analytics is different from common big data analytics.
− Data collection in the IoT comes at the cost of bandwidth, network,
energy and other resources.
− Data collection, delivery and processing is also depended on multiple
layers of the network.
− We need more resource-aware data analytics methods and cross-layer
− The solutions should work across different systems and multiple platforms
(Ecosystem of systems).
− Data sources are more than physical (sensory) observation.
− The IoT requires integration and processing of physical-cyber-social data.
− The extracted insights and information should be converted to a feedback
and/or actionable information.
IET sector briefing report
Available at: http://www.theiet.org/sectors/built-environment/resources/digital-technology.cfm
Other challenges and topics that I didn't talk about
Trust, resilience and
Networks, test-beds and
Applications and use-case