The tutorial was given in DEBS'2017, Barcelona, June 20, 2017

1. Speaker: Opher Etzion
Event Processing within the Human
Body – Tutorial

2. 2
Event processing
within the human body
may, in time, be the
most important use of
this technology and
the Internet of
Things in general

3. 3
In the human body, we sense, make sense of what
we sense, make constant decisions and carry
them out. Autonomous systems do the same.
When the domain is the human body itself it’s
getting interesting….
Sensing
Making
sense
from the
sensing
Real-time
decision
making
Acting
In this tutorial we’ll talk about
current and future applications
of IoT and autonomous systems
within the human body

4. OUTLINE
Background: events, autonomous systems, Body
Area Networks
Medical oriented applications of event
processing within the human body
Other applications of event processing within the
human body
The future perspective
4
Topic I
TOPIC II
Topic III
Topic IV

5. OUTLINE
Background: events, autonomous systems, Body
Area Networks
Medical oriented applications of event
processing within the human body
Other applications of event processing within the
human body
The future perspective
5
Topic I
TOPIC II
Topic III
Topic IV

6. None of the
authorized drivers
location is near the
car’s location
theft is concluded
Use a built-in car
stopper to slow the
intruder and dispatch
the security company
A person enters a
car and the car
starts moving;
the person does not
look like one of the
authorized drivers
Such applications
become possible
since everything is
connected
6

7. 7
The term “Internet of Things” was coined by Kevin
Ashton in 1999.
His observation was that all the data on the
Internet has been created by a human.
His vision was: “we need to empower computers with
their own means of gathering information, so they
can see, hear, and smell the world by themselves”.

8. 8
The world of sensors
1 Acoustic, sound, vibration
2 Automotive, transportation
3 Chemical
4 Electric current, electric potential,
magnetic, radio
5 Environment, weather, moisture,
humidity
6 Flow, fluid velocity
7 Ionizing radiation, subatomic particles
8 Navigation instruments
9 Position, angle, displacement, distance,
speed, acceleration
10 Optical, light, imaging, photon
11 Pressure
12 Force, density, level
13 Thermal, heat, temperature
14 Proximity, presence

9. 9
The value of sensors
Kevin Ashton: “track and count everything, and
greatly reduce waste, loss, and cost. We could
know when things needs replacing, repairing or
recalling, and whether they were fresh or past
their best”
The value is in the ability to know and react in a
timely manner to situations that are detected by
sensors

10. 10
Differences between the traditional Internet to
the Internet of Things
Topic Traditional Internet Internet of Things
Who creates content? Human Machine
How is the content
consumed?
By request By pushing information
and triggering actions
How content is
combined?
Using explicitly defined
links
Through explicitly
defined operators
What is the value? Answer questions Action and timely
knowledge
What was done so far? Both content creation
(HTML…) and content
consumption (search
engines)
Mainly content creation

11. Two separate but connected goals:
Awareness and Reaction
Awareness Reaction
Event
Detect Derive Decide Do

12. Detect
Something
that may be
important
The act of bringing into a system’s sphere of understanding knowledge
about an event.
The detection is done by sensors, instrumentation and human
reports.
SwimLane
TriggerEvent
Activity
State
Change

13. 13
Intelligent Detection
Determine what is actually been sensed:
vision understanding, voice understanding,
text understanding.

14. Derive
The act of becoming aware of events that are not directly detectable
by bringing together events with other events, data, patterns and
publishing the observation as a derived event.
Raw
events
Raw
events
Raw
events
A Person or a computer recognizes the pattern and
enters the
derived event or just reacts to it directly.

15. 15
Event processing: making sense from what we
sense…
Combining data from multi-sensors to get
observations, alerts, and actions in real-time gets
us to the issue of detecting patterns in event
streams

16. 16
Intelligent derivation
Find the causality between events and
situations.

17. Decide
The act of determining the course of action to do in response to the
situation. This includes the background information needed to be
collected to make the decision.
Pass through: Sometimes there is no decision. There is only
one course of action.

18. 18
Intelligent Decision
Finding the best decision some times under
real-time constraints may require an
intelligent process.

19. Do
The act of performing the course of action that was decided upon.
Notification: Sending a signal of sort to either a person or
system. This would include calling a web-service or
subscription to alerts.

20. 20
Intelligent Actions
Intelligent actuators

21. 21
Knowledge acquisition for IoT based systems
How do we know how to
make sense of all
these data?

22. 22
Contemporary
healthcare trends:
Telehealth

23. 23
Contemporary healthcare trends: Personalized Medicine

24. 24
Contemporary healthcare trends: Sensors and actuators

25. 25
Body Area Networks

26. 26
Body Area Networks
Source: Min Chen et al – Body Area Networks: A
Survey. Mobile Networks Applications, 16: 171-193,
2011.

27. 27
Body Area Networks
Source: Min Chen et al – Body Area Networks: A
Survey. Mobile Networks Applications, 16: 171-193,
2011.

28. OUTLINE
Background: events, autonomous systems, Body
Area Networks
Medical oriented applications of event
processing within the human body
Other applications of event processing within the
human body
The future perspective
28
Topic I
TOPIC II
Topic III
Topic IV

29. 29
Soldier in Battle (including: policeman, fire fighter…)
Wearables:
Camera
GPS
Biometric
Sensors
Goal:
Warning
Orientation
Assessment of
readiness by
commander.

30. 30
Wearables:
Camera
Motion sensors
Biometric
SensorsGoal:
Self feedback
Trainer feedback
Real-time game
management
for the coach.
Training and Sport

31. 31
Fitbit Surge:
Wristwatch that
contains GPS,
tracks steps,
monitors heart-
rate, connects
to community
Training monitoring

32. 32
NICU: Pre-mature babies monitoring
Personalized alerts based on collection of
monitors: when nurse should be alerted, when
physician should be alerted.
There are many false alerts that are ignored,
Missing or ignored alert is sometimes fatal

33. 33
iFeel Labs:
Remove stress by
bio-feedback.
Using games.
Stress control

34. 34
Goal:
Monitor and
diagnose sleep
disorders .
Sleep disorders monitoring
Source: N. Oliver & F. Flores-Mangas: Health
Gear – Automatic Sleep Apnea Detection and
Monitoring with Mobile Phone. Journal of
Communication, 2(2), 2007

35. 35
Goal:
Monitor and
diagnose sleep
disorders .
Sleep disorders monitoring
Source: N. Oliver & F. Flores-Mangas: Health
Gear – Automatic Sleep Apnea Detection and
Monitoring with Mobile Phone. Journal of
Communication, 2(2), 2007

36. 36
Pebble time:
Wrist watch that
measures sleep,
sleep patterns,
finds the best
time to wake
up.
Sleep control

37. 37
Wearables:
Wristband, chest
patch and
spirometer –
Monitoring both
the person and
the
environment
for allergy
factors
Goal:
Prevent, predict,
monitor asthma
attacks .
Monitoring Asthma Patients
Source:
https://news.ncsu.edu/2016/06/wearable-
tech-asthma-2016/

38. 38
Sensors and
actuators –
technology still
under
development
Goal:
to improve insulin
replacement
therapy until
glycemic
control is
practically
normal, and to
ease the
burden of
therapy for the
insulin-
dependent.
Implants for diabetes patients
Source:
https://www.slideshare.net/energexsystems/p
ancreas-presentation

39. 39
Implantable
cardioverter
defibrillators
(ICDs) are used
to detect
dangerously
fast
heartbeats and
give a lifesaving
shock to
correct the
heart’s rhythm.
A person with
an ICD has the
equivalent of a
paramedic
sitting on his
shoulder,
always watching
and ready to
give the heart
“the paddles,”
Implants for cardiovascular diseases
Source: http://www.webmd.com/heart-
disease/implantable-cardioverter-defibrillator

40. 40
Smart pacemaker
A pacemaker is a
small device
that's placed in
the chest or
abdomen to
help control
abnormal heart
rhythms. This
device uses
electrical
pulses to
prompt the
heart to beat
at a normal
rate.
Implants for cardiovascular diseases
source:
https://www.cambridgeconsultants.com/media
/press-releases/setting-pace

41. 41
Monitoring for
cancer cells,
replacing biopsy
tests.
Monitoring for
impact of
treatments and
medications for
personalized
treatment.
In the future:
also actuators
that serve to
treat and
eliminate
Implants for cancer treatment

42. 42
Closed loop to
monitor and
regulate DBS
Helping
Essential
Tremor and
Parkinson
Patients.
Controlling deep brain stimulation

43. 43
Connecting audio
signals to
hearing nerves.
Implants that bypass damaged
senses
https://www.foundationforsightandsound.org/wo
rdpress/?page_id=636

44. 44
Aiding blind
people
Implants that bypass damaged senses

45. 45
OrCam MyEye is a tiny camera that uses recognition
technology to help the blind and visually impaired
identify people, text and things. The portable, intuitive
device is comprised of a small camera that can be
discreetly affixed to the wearer’s own glasses, plus a
lightweight computer processor that’s small enough to
be carried in a pocket. OrCam MyEye works by
capturing an image of an object and translating it into
audio for the wearer. The device is able to read text,
identify products, read street signs and even
recognize faces.
Implants that bypass damaged senses

46. 46
Almost the entire electromagnetic spectrum is invisible
to the human eye. Hyper imaging technology and AI
will provide super human vision that will help to spot
dangers that are hidden from the view
Source: IBM Predictions for 2022.
Super human vision

47. 47
AI based diagnostic of mental disorders based on
behavior, speech, writing etc…
Source: IBM Predictions for 2022.
Detection of mental disorders

48. 48
Personalized aides for elderly to maintain independent life
Motion
sensor
Door
sensor
Chair
Sensor
Voice
Sensor
Alert
family
member
Alerts example:
Door was not locked within 2 minutes after entrance
Falling event detected
Vocal distress detected
No motion for certain time period detected
While much technology exists, it is not widely
used. It needs to be more personalized, more
affordable, and much simpler…
The research required is multi-disciplinary:
Technology oriented, human oriented, economic oriented and particular
domain oriented

49. 49
Family planning and
contraceptives based on
wristband that measures and
systems that accumulates
personal data

50. OUTLINE
Background: events, autonomous systems, Body
Area Networks
Medical oriented applications of event
processing within the human body
Other applications of event processing within the
human body
The future perspective
50
Topic I
TOPIC II
Topic III
Topic IV

51. 51
Implant in the
brain retrieves
electric signals.
Brain computer interface
Source: Ed Grabaianowski: How Brain-Computer
Interfaces Works:
http://computer.howstuffworks.com/brain-
computer-interface.htm

52. 52
Brain computer interface
Source: Ed Grabaianowski: How Brain-Computer
Interfaces Works:
http://computer.howstuffworks.com/brain-
computer-interface.htm

53. 53
Microchip
implanted in
the body to
identify
persons.
Identification implants
Applications for
security, open
doors, payment
(activating
credit cards)
presence and
more.

54. 54
Next generation
of glasses that
connect to the
Internet,
identify
persons,
interface for
autonomic
assitant
Seeing Virtual

55. 55
Automated
personal
assistant
Sensors that determine the context are linked to
active advisors. They understand your context and
even listen to your conversations and give you
suggestions of what to say (e.g. through google
glass).
2018

56. 56
Chemical plants safety
Monitoring people movement and compliance with
safety regulations, leak of chemical materials…

57. 57
Summary: Detect-Derive-Decide-Do in the human
body level
Detect –
Wearables +
implants
Derive -
knowledge +
learning +
personalization
Decide – AI
+ decision
science
Do –
Actuators
:
autonomic
and
manual

58. OUTLINE
Background: events, autonomous systems, Body
Area Networks
Medical oriented applications of event
processing within the human body
Other applications of event processing within the
human body
The future perspective
58
Topic I
TOPIC II
Topic III
Topic IV

59. 59
The vision: Human Sapiens 2.0 –
next phase of the evolution

60. 60
Reprogramming
the body to
eliminate aging.
Eternal Life?

61. 61
Kurzweil’s time
Stamp for
singularity.
Eternal Life?

62. 62

63. 63
Source: S.A. Dash – Mind Uploading
https://www.slideshare.net/prabin210/mind-
uploading

64. 64
Lung on a chip

65. 65
Arthery on a chip

66. 66
Human body
With multiple organs
on a chip

67. 67
Microsoft Biological Computation: Goal to program the body in order to
eliminate cancer
By treating cancer like an
information processing
system, Microsoft
researchers are able to adapt
tools typically used to model
computational processes to
model biological ones.
Ultimately, the company hopes
to create molecular computers
to program the body to fight
cancer cells immediately after
detection
Andrew PhillipsSource: Christina Farr, Mind and Machine,
September 2016,
https://www.fastcompany.com/3063835/micr
osofts-biological-computing-lab-wants-to-
fight-diseases-by-reprogrammin

68. 68
Biological computers destroy cancer
cells

69. 69
Hacking the DNA – programming the cells
CELLS ARE BASICALLY tiny
computers: They send and
receive inputs and output
accordingly. If you chug a
Frappuccino, your blood sugar
spikes, and your pancreatic
cells get the message. Output:
more insulin.
A cell could be programmed,
for example, with a so-called
NOT logic gate. This is one of
the simplest logic
instructions: Do NOT do
something whenever you
receive the trigger…. Biologist
Wilson Wong of Boston
University, who led the
research, refers to these
engineered cells as “genetic
circuits.”
Applications: diagnostics,
disease fighting, in the
future: hacking the aging
mechanism.
Source: Sophia Chen, Wired, March 2017
https://www.wired.com/2017/03/biologists-
made-logic-gates-dna/

70. 70
Biological molecules are used for storing information and transmitting it
to other cells
“For example, suppose we
could partner with microbes
and plants to record events,
natural or otherwise, and
convert this information into
easily observed signals. That
would greatly expand our
ability to monitor the
environment,” Drew Andy,
Stanford University

71. 71
Privacy considerations
The traditional Internet and social networks are already
compromising privacy in the virtual world
Wearable devices and implants expose a lot of information about
humans. There are already debates about biometric stores.

72. 72
Security considerations
Murder by the Internet
“With so many devices being Internet connected, it makes murdering
people remotely relatively simple, at least from a technical
perspective. That’s horrifying,” said IID president and CTO Rod
Rasmussen. “Killings can be carried out with a significantly lower
chance of getting caught, much less convicted, and if human history
shows us anything, if you can find a new way to kill, it will be
eventually be used.”
EXAMPLES: Turn off pacemakers, Shutdown car systems while
driving, stop IV drip from functioning

73. 73
Dangers and challenges
Confusing a sensor
Changing the rules of the game
Abusing an actuator

74. 74
Confusing a sensor
The same as confusing the human
eyes.
See things that don’t exist, don’t
see things that exist, distort
picture…

75. 75
Confusing a sensor
Can be used to sabotage, to
commit crimes, frauds and more.

76. 76
Confusing a sensor
Example from another domain: the
Twitter hoax

77. 77
Changing the rules of the game
The logic is
available.
The ease of
modification
can be
abused to
add/delete/
modify rules,
change
thresholds…

78. 78
Changing the rules of the game
Changing data relevant for the
system: maps, pictures, person’s
data…

79. 79
Abusing actuators
Deviating
from
course,
shutting
down,
activating in
wrong
mode…

80. 80
The Solaria effect
||machines|| >> ||men||

81. 81
The Driverless Car
Dilemma as an example
to theautonomous
computing dilemmas.

82. 82
The Deep Learning
Dilemma
Autonomous systems employ deep learning
techniques for self-improvement. The
designer loses control over the behavior.

83. 83
Who will move to the next phase of evolution?

84. 84
Social Implications
Spreading the progress or keeping it at
the hands of small fraction of the
population

85. 85
Social Implications
Question: What are the implications
of long life?

86. 86
Social Implications
Question: What are the implications
of Homo Sapiens 2.0 with embedded
computers in the brain and body?

87. 87
Processing human related events is one
of the enabling and essential
technologies to the future, and might
be the area where event processing
eventually will have the most impact.