Invited talk, University of York, November 2016

1. Internet of Things and Data Analytics
for Smart Cities and eHealth
1
Payam Barnaghi
Institute for Communication Systems (ICS)/
5G Innovation Centre
University of Surrey
Guildford, United Kingdom
University of York, November 2016

2. “A hundred years hence people will be so avid
of every moment of life, life will be so full of
busy delight, that time-saving inventions will
be at a huge premium…”
“…It is not because we shall be hurried in
nerve-shattering anxiety, but because we shall
value at its true worth the refining and restful
influence of leisure, that we shall be impatient
of the minor tasks of every day….”
The March 26, 1906, New Zealand Star :
Source: http://paleofuture.com

3. 3
IBM Mainframe 360, source Wikipedia

4. Apollo 11 Command Module (1965) had
64 kilobytes of memory
operated at 0.043MHz.
An iPhone 5s has a CPU running at speeds
of up to 1.3GHz
and has 512MB to 1GB of memory
Cray-1 (1975) produced 80 million Floating
point operations per second (FLOPS)
10 years later, Cray-2 produced 1.9G FLOPS
An iPhone 5s produces 76.8 GFLOPS – nearly
a thousand times more
Cray-2 used 200-kilowatt power
Source: Nick T., PhoneArena.com, 2014
image source: http://blog.opower.com/

5. Computing Power
5
−Smaller size
−More Powerful
−More memory and more storage
−"Moore's law" over the history of computing, the
number of transistors in a dense integrated circuit
has doubled approximately every two years.

6. Smaller in size but larger in scale
6

7. The old Internet timeline
7Source: Internet Society

8. Connectivity and information exchange was
(and is) the main motivation behind the
Internet; but Content and Services are now
the key elements;
and all started growing rapidly by the
introduction of the World Wide Web (and
linked information and search and discovery
services).
8

9. Early days of the web
9

10. Search on the Internet/Web in the early days
10

11. Source: Intel, 2012

12. Source: http://www.techspartan.co.uk

13. 13P. Barnaghi et al., "Digital Technology Adoption in the Smart Built Environment", IET Sector Technical Briefing, The Institution of Engineering and Technology
(IET), I. Borthwick (editor), March 2015.

14. 14
Sensor devices are becoming widely available
- Programmable devices
- Off-the-shelf gadgets/tools

15. Internet of Things: The story so far
RFID based
solutions
Wireless Sensor and
Actuator networks
, solutions for
communication
technologies, energy
efficiency, routing, …
Smart Devices/
Web-enabled
Apps/Services, initial
products,
vertical applications, early
concepts and demos, …
Motion sensor
Motion sensor
ECG sensor
Physical-Cyber-Social
Systems, Linked-data,
semantics, M2M,
More products, more
heterogeneity,
solutions for control and
monitoring, …
Future: Cloud, Big (IoT) Data
Analytics, Interoperability,
Enhanced Cellular/Wireless Com.
for IoT, Real-world operational
use-cases and Industry and B2B
services/applications,
more Standards…

16. Data- Challenges
− Multi-modal and heterogeneous
− Noisy and incomplete
− Time and location dependent
− Dynamic and varies in quality
− Crowed sourced data can be unreliable
− Requires (near-) real-time analysis
− Privacy and security are important issues
− Data can be biased- we need to know our data!
16

17. Speed of light?
17
Image source: The Brain with David Eagleman, BBC

18. Device/Data interoperability
18
The slide adapted from the IoT talk given by Jan Holler of Ericsson at IoT Week 2015 in Lisbon.

19. WoT/IoT
WSN
WSN
WSN
WSN
WSN
Network-enabled
Devices
Semantically
annotate data
19
Gateway
CoAP
HTTP
CoAP
CoAP
HTTP
6LowPAN
Semantically
annotate data
http://mynet1/snodeA23/readTemp?
WSN
MQTT
MQTT
Gateway
Gateway

20. 20
Some good existing models: W3C SSN Ontology
Ontology Link: http://www.w3.org/2005/Incubator/ssn/ssnx/ssn
M. Compton, P. Barnaghi, L. Bermudez, et al, "The SSN Ontology of the W3C Semantic Sensor Network Incubator Group", Journal of Web Semantics,
2012.

21. IoT-lite ontology
21

22. Spatial Data on the Web WG
https://www.w3.org/2015/spatial/charter

23. 23

24. Hyper/CAT
24
Source: Toby Jaffey, HyperCat Consortium, http://www.hypercat.io/standard.html
- Servers provide catalogues of resources to
clients.
- A catalogue is an array of URIs.
- Each resource in the catalogue is annotated
with metadata (RDF-like triples).

25. FIWARE IoT Discovery Generic Enabler
25http://catalogue.fiware.org/enablers/iot-discovery/documentation

26. New Generation of Search Engines
26
P. Barnaghi, A. Sheth, “On Searching the Internet of Things: Requirements and Challenges”, to appear, IEEE Intelligent Systems, 2016.

27. On Searching the Internet of Things
27
P. Barnaghi, A. Sheth, “On Searching the Internet of Things: Requirements and Challenges”, to appear, IEEE Intelligent Systems, 2016.

28. A discovery engine for the IoT
28A. HosseiniTabatabaie, P. Barnaghi, C. Wang, L. Dong, R. Tafazolli, "Method and Apparatus for Scalable Data Discovery in IoT Systems”, US
Patents, CNV12174, May 2014.
Let’s assume that attribute x has an
alphabet Ax ={ax1,…,axs}. Query for
a data item (q) that is described
with attributes x, y and z, is then
represented as q={x=axk & y=ayl &
z=azm}
The average ratio of matching
processes that are required to
resolve this query at n:

29. A GMM model for indexing
29
Average Success rates
First attempt: 92.3% (min)
At first DS: 92.5 % (min)
At first DSL2 : 98.5 %
(min)
Number of attempts
Percentageofthetotalqueries
A. HosseiniTabatabaie, P. Barnaghi, C. Wang, L. Dong, R. Tafazolli, "Method and Apparatus for Scalable Data Discovery in IoT Systems”, US
Patents, CNV12174, May 2014.

30. Indexing spatial data with multiple
attributes
30
Fathy Y., Barnaghi P., Tafazolli R., “Distributed in-network indexing mechanism for the Internet of Things (IoT)”, submitted to IEEE ICC 2017.
Fathy Y., Barnaghi P., Enshaeifar S., Tafazolli R., "A Distributed In-network Indexing Mechanism for the Internet of Things", IEEE World Forum on IoT, 2016.

31. Adaptive Clustering
31D. Puschmann, P. Barnaghi, R.Tafazolli, "Adaptive Clustering for Dynamic IoT Data Stream", IEEE Internet of Things Journal, 2016.

32. Adaptive clustering
32D. Puschmann, P. Barnaghi, R. Tafazolli, "Marginal Distribution Clustering of Multi-variate Streaming IoT Data",
IEEE World Forum on IoT, Dec. 2016.

33. Dynamic clusters
33D. Puschmann, P. Barnaghi, R. Tafazolli, "Marginal Distribution Clustering of Multi-variate Streaming IoT Data",
IEEE World Forum on IoT, Dec. 2016.

34. Dynamic clusters - multivariate data
34D. Puschmann, P. Barnaghi, R. Tafazolli, "Marginal Distribution Clustering of Multi-variate Streaming IoT Data",
IEEE World Forum on IoT, Dec. 2016.

35. Creating Patterns-
Adaptive sensor SAX
35
F. Ganz, P. Barnaghi, F. Carrez, "Information Abstraction for Heterogeneous Real World Internet Data”, IEEE Sensors Journal, 2013.

36. From SAX patterns to events/occurrences
36
F. Ganz, P. Barnaghi, F. Carrez, "Automated Semantic Knowledge Acquisition from Sensor Data", IEEE Systems Journal, 2014.

37. Learning ontology from sensory data
37

38. Patterns and Segmentation of Time-series data
38
A. Gonzalez-Vidal, P. Barnaghi, A. F. Skarmeta, BEATS: Blocks of Eigenvalues Algorithm for Time series Segmentation,
Submitted to IEEE TKDE, 2016.

39. KAT- Knowledge Acquisition Toolkit
F. Ganz, D. Puschmann, P. Barnaghi, F. Carrez, "A Practical Evaluation of Information Processing and Abstraction Techniques for the Internet of
Things", IEEE Internet of Things Journal, 2015.
39
https://github.com/CityPulse/Knowledge-Acquisition-Toolkit-2.0
http://kat.ee.surrey.ac.uk

40. KAT V.2.0
40

41. IoT data
41

42. Analysing social streams
42Collaboration with Wright State University:

43. City event extraction from social streams
43
Tweets from a city
POS
Tagging
Hybrid NER+
Event term
extraction
GeohashingGeohashing
Temporal
Estimation
Temporal
Estimation
Impact
Assessment
Impact
Assessment
Event
Aggregation
Event
AggregationOSM LocationsOSM Locations SCRIBE ontologySCRIBE ontology
511.org hierarchy511.org hierarchy
City Event ExtractionCity Event Annotation
P. Anantharam, P. Barnaghi, K. Thirunarayan, A.P. Sheth, "Extracting City Traffic Events from Social Streams", ACM Trans. on Intelligent Systems
and Technology, 2015.

44. CRF formalisation – for annotation
44
A General CRF Model

45. Extracted events and the ground truth
45Open source software: https://osf.io/b4q2t/

46. Extracting city events
46
City Infrastructure
Yes it is police @hasselager
… there directing traffic
CRF-
based
NER
Tagging Multi-view
Event
Extraction
Loc. Est. =
“hasselager,
aarhus”
Loc. Est. =
“hasselager,
aarhus”
Temp. Est. =
“2015-2-19
21:07:17”
Temp. Est. =
“2015-2-19
21:07:17”
Level = 2Level = 2
Event = TrafficEvent = Traffic
OSM
Loc.
OSM
Loc.
CrimeCrimeTransp.Transp.
City Event Extraction
CNN
POS+NER
Event term
extraction
CulturalCultural SocialSocial Enviro.Enviro. SportSport HealthHealth
DataData
Transp.Transp.
Yes <O> it <O> is <O> police <B-CRIME>
@hasselager <B-LOCATION>… <O> there <O>
directing <O> traffic <B-TRAFFIC>
Yes <S-NP/O> it <S-NP/O> is <S-VP/O> police
<S-NP/O> @hasselager <S-LOC> ... <O/O> there
<S-NP/O> directing <S-VP/O> traffic <S-NP/O>
Nazli FarajiDavar, Payam Barnaghi, "A Deep Multi-View Learning Framework for City Event Extraction from Twitter Data Streams", submitted to ACM Transactions
on Intelligent Systems and Technology (TIST), Nov. 2015.

47. Extracting city events
47
http://iot.ee.surrey.ac.uk/citypulse-social/
Nazli FarajiDavar, Payam Barnaghi, "A Deep Multi-View Learning Framework for City Event Extraction from Twitter Data Streams", submitted to ACM
Transactions on Intelligent Systems and Technology (TIST), Nov. 2015.

48. Cities of the future
48
http://www.globalnerdy.com/2007/08/28/home-electronics-of-the-future-as-predicted-28-years-ago/

49. 49
Source: BBC News

50. Source: The dailymail, http://helenography.net/, http://edwud.com/

51. What are smart cities?
51
“An ecosystem of systems enabled by the
Internet of Things and information
communication technologies.”
“People, resources, and information coming
together, operating in an ad-hoc and/or
coordinated way to improve city operations
and everyday activities.”

52. What does makes smart cities “smart”?

53. Smart Citizens (more informed and more in control)
Smart Governance (better services and informed decisions)
Smart Environment
Providing more equality and wider reach
Context-aware and situation-aware services
Cost efficacy and supporting innovation
What does makes smart cities “smart”?

54. How do cities get smarter?

55. How do cities get smarter?
55
Continuous (near-) real-time sensing/monitoring
and data collection
Linked/integrated data and linked/integrated
services
Real-time intelligence and actionable-information
for different situations/services
Smart interaction and actuation
Creating awareness and effective participation

56. How can technology help to make
cities smarter?

57. The role of data
57
Source: The IET Technical Report, Digital Technology Adoption in the Smart Built Environment: Challenges and opportunities of
data driven systems for building, community and city-scale applications,
http://www.theiet.org/sectors/built-environment/resources/digital-technology.cfm

58. 58
“Each single data item can be important.”
“Relying merely on data from sources that are
unevenly distributed, without considering
background information or social context, can
lead to imbalanced interpretations and
decisions.”
?

59. 59
“The ultimate goal is transforming the raw data
to insights and actionable information and/or
creating effective representation forms for
machines and also human users, and providing
automated services.”
This usually requires data from multiple sources,
(near-) real time analytics and visualisation
and/or semantic representations.

60. IoT environments are usually dynamic and (near-) real-
time
60
Off-line Data analytics
Data analytics in dynamic environments
Image sources: ABC Australia and 2dolphins.com

61. What type of problems we expect to solve
using the IoT and data analytics solutions?

62. 62Source LAT Times, http://documents.latimes.com/la-2013/
A smart City example
Future cities: A view from 1998

63. 63
Source: http://robertluisrabello.com/denial/traffic-in-la/#gallery[default]/0/
Source: wikipedia
Back to the Future: 2013

64. Common problems
64
Guildford, Surrey

65. 65

66. 101 Smart City scenarios
66http://www.ict-citypulse.eu/scenarios/
Dr Mirko Presser
Alexandra Institute
Denmark

67. Live data
67

68. 68
Event Visualisation

69. CityPulse demo
69

70. Users in control or losing control?
70
Image source: Julian Walker, Flicker

71. 71
http://www.ict-citypulse.eu/
https://github.com/CityPulse

72. eHealth
72
Ramesh Jain, Micro reports and Situation Recognition at social machines workshop, 2016.

73. 73
Ramesh Jain, Micro reports and Situation Recognition at social machines workshop, 2016.

74. Medical/Health Data
− The average person is likely to generate more than one
million gigabytes of health-related data in their lifetime. This is
equivalent to 300 million books.
− Medical data is expected to double every 73 days by 2020.
− 80% of health data is invisible to current systems because it’s
unstructured.
− Less than 50% of medical decisions meet evidence-based
standards. (source: The rand corporation)
74Source: IBM Research

75. Unstructured data!
Heterogeneity, multi-modality and volume are
among the key issues.
Often natural language!
We need interoperable and machine-interpretable
solutions…
75

76. Medical/Health decision making
− One in five diagnoses are incorrect or incomplete and nearly
1.5 million medication errors are made in the US every year.
− Medical journals publish new treatments and discoveries
every day.
− The amount of medical information available is doubling every
five years and much of this data is unstructured - often in
natural language.
− 81 percent of physicians report that they spend five hours per
month or less reading journals.
76Source: IBM Research

77. Medical/Health data in decision making
− Patient histories can give clues.
− Electronic medical record data provide lots of information.
− Current observation and measurement data and fast analysis
of the data can help (combined with other data/medical
records).
− This needs fast/accurate/secure data:
− Collection/retrieval
− Communication
− Sharing/Integration
− Processing/Analysis
− Visualisation/presentation
77

78. IBM Watson
78
Watson can process the patient data to find
relevant facts about family history, current
medications and other existing conditions.
It can combines this information with current
findings from tests and instruments and then
examines all available data sources to form
hypotheses and test them.
Watson can also incorporate treatment guidelines,
electronic medical record data, doctor's and
nurse's notes, research, clinical studies, journal
articles, and patient information into the data
available for analysis.
Source: IBM
Watson can read 40 million documents in 15 seconds.

79. Sensely
79
Source: http://sense.ly/

80. Healthcare data analytics- Symptom management
80N. Papachristou, C. Miaskowski, P. Barnaghi, R. Maguire, N. Farajidavar, B. Cooper and X. Hu,
"Comparing Machine Learning Clustering with Latent Class Analysis on Cancer Symptoms’ Data", IEEE-NIH 2016, Nov. 2016.

81. Technology Integrated Health Management (TIHM)
− An Internet of Things testbed to support dementia patients
and their carers/doctors.
− For patients with early to mild dementia
− Remote and technology assisted care, monitoring and alert.
81

82. Innovation Partners
Nine companies with 25+ devices and services, including monitors, sensors,
apps, hubs, virtual assistants, location devices and wearables

83. The Health Challenge: Dementia
 16,801 people with dementia in Surrey – set to rise to 19,000
by 2020 (estimated) - nationally 850,000 - estimated 1m by
2025 (Alzheimer’s Society)
 Estimated to cost £26bn p/a in the UK (Alzheimer’s Society):
health and social care (NHS and private) + unpaid care
 Devices in the IoT will provide actionable data on agitation,
mood, sleep, appetite, weight loss, anxiety and wandering – all
have a big impact on quality of life and wellbeing

84. The Health Challenge: Falls
 Surrey spends £10m a year on fracture care – with 95% of hip
fractures caused by falls
 People with dementia suffer significantly higher fall rates that
cause injury – with falls the most common cause of injury-
related deaths in the over-75s
 Devices in the IoT will monitor location, activity and incident,
supporting health/care staff and carers, enabling early
intervention

85. The Health Challenge: Carers
 5.4m carers supporting ill, older or disabled family members,
friends and partners in England - expected to rise by 40%
over the next 20 years.
 Value of such informal care estimated at £120bn a year – but
carer ‘burnout’ a key reason why loved ones require
admission to a care/nursing home.
 Devices in the IoT will support carers in their caring asks –
and support their own health and wellbeing.

86.  Infrastructure
 Interoperability, integration
 Security
 Data governance
 Scalability
Technical Challenge

87. Device/Data interoperability
87

88. FIHR4TIHM
88

89. Gateway
Gatewa
y
Data Analytics
Engine
IoT Test Bed Cloud
External NHS, GP IT systems
Possible links to
Other Test Beds
HyperCat
Gateway
HyperCat
HyperCat
HyperCat
Data-driven and patient
centered Healthcare
Applications

91.  Extend into homes – year
1 via two CCG areas,
rolling out across four
more CCGs in year 2
 Reach 350 homes – with a
control group of 350 – via
dementia register
 Focus on most effective
product combinations –
with potential for more
via an open call
Roll Out
NE Hants & Farnham
Living Lab
Guildford
& Waverley
Rest of Surrey
And beyond…

92. In Conclusion
− Lots of opportunities and in various application domains;
− Enhanced and (near-) real-time insights;
− Supporting more automated decision making and in-depth
analysis of events and occurrences by combining various
sources of data;
− Providing more and better information to citizens;
− Citizens in control;
− Transparency and data management issues (privacy, security,
trust, …);
− Reliability and dependability of the systems.
92

93. Accumulated and connected knowledge?
93
Image courtesy: IEEE Spectrum

94. Other challenges and topics that I didn't talk about
Security
Privacy
Trust, resilience and
reliability
Noise and
incomplete data
Cloud and
distributed computing
Networks, test-beds and
mobility
Mobile computing
Applications and use-case
scenarios
94

95. Q&A
− Thank you.
http://personal.ee.surrey.ac.uk/Personal/P.Barnaghi/
@pbarnaghi
p.barnaghi@surrey.ac.uk