DEBS 2015 tutorial When Artificial Intelligence meets the Internet of Things

1. When Artificial Intelligence meets the Internet of Things. DEBS’15 tutorial Speaker: Opher Etzion

2. 2 The autonomous car Needs sensors for observing what happens now, needs intelligence to understand what it observes, needs intelligence to drive, needs actuators to carry out the driving….

3. 3 Like the human body, we need to sense, to make sense of what we sense, to make constant decisions and to carry them out. Sensing Making sense from the sensing Real-time decision making Acting

4. OUTLINE A quick intro to IoT and its relationship with AI Some applications of Intelligent IoT The AI perspective The future perspective 4 Topic I TOPIC II Topic III Topic IV

6. None of the authorized drivers location is near the car’s location theft is concluded Use a built-in car stopper to slow the intruder and dispatch the security company A person enters a car and the car starts moving; the person does not look like one of the authorized drivers Such applications become possible since everything is connected 6

7. 7 The term “Internet of Things” was coined by Kevin Ashton in 1999. His observation was that all the data on the Internet has been created by a human. His vision was: “we need to empower computers with their own means of gathering information, so they can see, hear, and smell the world by themselves”.

8. 8 The world of sensors 1 Acoustic, sound, vibration 2 Automotive, transportation 3 Chemical 4 Electric current, electric potential, magnetic, radio 5 Environment, weather, moisture, humidity 6 Flow, fluid velocity 7 Ionizing radiation, subatomic particles 8 Navigation instruments 9 Position, angle, displacement, distance, speed, acceleration 10 Optical, light, imaging, photon 11 Pressure 12 Force, density, level 13 Thermal, heat, temperature 14 Proximity, presence

9. 9 The value of sensors Kevin Ashton: “track and count everything, and greatly reduce waste, loss, and cost. We could know when things needs replacing, repairing or recalling, and whether they were fresh or past their best” The value is in the ability to know and react in a timely manner to situations that are detected by sensors

10. 10 Differences between the traditional Internet to the Internet of Things Topic Traditional Internet Internet of Things Who creates content? Human Machine How is the content consumed? By request By pushing information and triggering actions How content is combined? Using explicitly defined links Through explicitly defined operators What is the value? Answer questions Action and timely knowledge What was done so far? Both content creation (HTML…) and content consumption (search engines) Mainly content creation

11. Two separate but connected goals: Awareness and Reaction Awareness Reaction Event Detect Derive Decide Do

12. Detect Some Noun of Importance but different The act of bringing into a system’s sphere of understanding knowledge about an event. The detection is done by sensors, instrumentation and human reports. SwimLane TriggerEvent Activity State Change

13. 13 Intelligent Detection Determine what is actually been sensed: vision understanding, voice understanding, text understanding.

14. Derive The act of becoming aware of events that are not directly detectable by bringing together events with other events, data, patterns and publishing the observation as a derived event. Raw events Raw events Raw events A Person or a computer recognizes the pattern and enters the derived event or just reacts to it directly.

15. 15 Event processing: making sense from what we sense… Combining data from multi-sensors to get observations, alerts, and actions in real-time gets us to the issue of detecting patterns in event streams

16. 16 Intelligent derivation Find the causality between events and situations. We discuss the notion of causality later.

17. Decide The act of determining the course of action to do in response to the situation. This includes the background information needed to be collected to make the decision. Pass through: Sometimes there is no decision. There is only one course of action.

18. 18 Intelligent Decision Finding the best decision some times under real-time constraints may require an intelligent process.

19. Do The act of performing the course of action that was decided upon. Notification: Sending a signal of sort to either a person or system. This would include calling a web-service or subscription to alerts.

20. 20 Intelligent Actions Intelligent actuators

21. 21 Knowledge acquisition for IoT based systems How do we know how to make sense of all these data?

23. 23 IoT and robotics Robots serve as intelligent actuators

24. 24 Healthcare robotics Rehabilitation robots: enhancing patients with motoric and cognitive skills Assistive robots: Robots for independent living of disabled persons

25. 25 Some future healthcare robotics applications Automated assistance of monitored patients

26. 26 Some future healthcare robotics applications Help in sit-to-stand and sit-down actions for people with motor disabilities

27. 27 Some future healthcare robotics applications Autonomous moving of drugs and medical equipment within the hospital

28. 28 Some future healthcare robotics applications Support of medical staff in various activities

29. 29 Some future healthcare robotics applications People movement and movement monitoring

30. 30 Some future healthcare robotics applications People assistance in panic and danger situations

31. 31 I( The classical use of robots are for industrial purposes: production, machinery control, product design… Industrial Robots

32. 32 I( Autonomic management and coordination of production activities among multiple robots Industrial Robots and IoT

33. 33 I( Autonomous management of equipment and instruments Industrial Robots and IoT

34. 34 I( Immediate reaction to critical situations such as: high temperature, harmful chemicals in the air Industrial Robots and IoT

35. 35 I( Autonomic control of electrical and energy plants Industrial Robots and IoT

36. 36 Robotics for defense Robots are used for unmanned tools (ground and air) for transport and intelligence , threat detection and combat

37. 37 Robotics for defense and IoT Autonomous and smart detection of harmful chemicals and biological weapons

38. 38 Robotics for defense and IoT Autonomic control of land vehicles and aircrafts

39. 39 Robotics for defense and IoT Identification and access prevention of suspicious people intruding to sensitive places

40. 40 Robotics for defense and IoT Rescue trapped people

41. 41 The Internet of things for the elderly and healthcare in general

42. 42 Safety sensors Motion sensor Door sensor Chair Sensor Voice Sensor Alert family member Alerts example: Door was not locked within 2 minutes after entrance Falling event detected Vocal distress detected No motion for certain time period detected

43. 43 Medical sensors for the elderly

44. 44 E-Health sensors Personalized alerts based on collection of monitors

45. 45 Pre-mature babies monitoring Personalized alerts based on collection of monitors: when nurse should be alerted, when physician should be alerted. There are many false alerts that are ignored, Missing or ignored alert is sometimes fatal

46. 46 Track everything in a hospital

47. 47 Track the progress of a surgery relative to the plan Detect significant deviation from plan that requires rescheduling and trigger real-time rescheduling of surgeries, assignments, and equipments.

48. 48 Dynamic planning Example: traffic control; patient treatment; serviceman scheduling

49. 49 AI meets IoT – Apple’s Perspective Siri was released as Apple’s “intelligent personal assistant”. A sensor enabled Siri is targeted as a “smart home solution”

50. 50 AI meets IoT – Google’s Perspective Google acquired a collection of IoT related companies and then acquired AI company DEEPMIND that uses Neural Nets and Reinforcement learning. The aim is to develop a machine with intelligence of a toddler with IoT providing sensing capabilities

51. 51 AI meets IoT – Facebook’s Perspective Facebook acquires wit.ai – a speech recognition company. Making the Internet of Things voice controlled.

53. 53 Vision understanding Robocup tournament: Robots playing football. Strong vision capabilities are required.

54. 54 Vision understanding Tracking objects over time from a collection of cameras

55. 55 Vision understanding Grace, the robot, can communicate with her surrounding, understand gestures, attended conferences, understands that she had to stand in a line, go in an elevator and ask people to press the floor number…

56. 56 Speech recognition Acoustic analysis, linguistic Interpretation

57. 57 Causality In order to derive situations from events there is a need to identify causalities. Statistical methods can infer correlations. Causality inference is more tricky….

58. 58 Causalities in events Type I: predetermined causality - Event E2 always (or conditionally) occurs as a result the occurrence of E1, thus we don't need to have any sensor to detect event E2 we may assume it happened if E1 happened (and the condition is satisfied), some time offset or interval may be attached to this causality. Note that in this case E1 and E2 are both raw events. Necessity and relevance

59. 59 Causalities in events Type II: The event E1 is an input to a processing element PE and event E2 is an output of PE. In this case E2 is a derived (virtual) event. The specification of PE is part of the system, thus the context and conditions are known. Necessity and relevance

60. 60 Causalities in events Type III: The event E1 is an event that is sent from a computerized system to a consumer C. C applies (conditionally) some action AC, where the specification of AC is not known to us, but we observe that it emits the event E2. This is another type of causality (the event E2 would not have been emitted, if E2 would not have triggered AC), however, E2 may or may not have functional dependency with respect to E1 Necessity? and relevance?

61. 61 Causal inference How the knowledge about causality is being acquired? Expert knowledge Statistical inference Inference using semantic or association net Necessity? and relevance?

62. 62 Dangers of using correlation as causality indicator Correlation between A and B: 1. A causes B 2. B causes A 3. There is C which causes both A and B 4. A combination of all three interpretations The faster windmills are observed to rotate, the more wind is observed to be. Therefore wind is caused by the rotation of windmills.

63. 63 Dangers of using correlation as causality indicator Correlation between A and B: 1. A causes B 2. B causes A 3. There is C which causes both A and B 4. A combination of all three interpretations Sleeping with one's shoes on is strongly correlated with waking up with a headache. Therefore, sleeping with one's shoes on causes headache. (correct answer: going to bad drunk causes both)

64. 64 Dangers of using correlation as causality indicator Correlation between A and B: 1. A causes B 2. B causes A 3. There is C which causes both A and B 4. A combination of all three interpretations As ice cream sales increase, the rate of drowning deaths increases sharply. Therefore, ice cream consumption causes drowning. (real answer: they are both in the same context – summer).

65. Temporal indeterminacy Inexact indicator Probability Event did not occur 0.4 Event occurred before T1 0.1 Event occurred in [T1, T2] 0.45 Event occurred after T2 0.05 T1 T2

66. False positives and negatives False positive: The pattern is matched; The real-world situation does not occur False negative: The pattern is not matched; The real-world situation occurs Learning from experience

67. 67 Data is not good enough…

68. 68 Real-time decision under uncertainty Robust RT Optimization Stochastic RT Optimization Simulation-based RT optimization

69. Handling event uncertainties Uncertain whether an reported event has occurred (e.g. accident) Uncertain what really happened. What is the type and magnitude of the accident (vehicles involved, casualties) Uncertain when an event occurred (will occur): timing of forecasted congestion Uncertain where an event occurred (will occur): location of forecasted congestion Uncertain about the level of causality between a car heading towards highway and a car getting into the highway Uncertain about the accuracy of a sensor input: count of cars, velocity of cars… The pattern: more than 100 cars approach an area within 5 minutes after an accident derives a congestion forecasting Uncertain about the validity of a forecasting pattern Uncertain about the quality of the decision about traffic lights setting

70. Predictive Event Processing (1) VS. Photo by Michael Gray, Flickr

71. Predictive Event Processing (2) VS. +

72. Predictive Event Patterns Pattern  Future event, probability, time interval “4 high value deposits from different geographic locations within 3 days”  “0.6 chance for a large transfer abroad, in 1 day” “Output event will occur with distribution D over interval (t1,t2)” Stock decrease of > 5% in 3 hours  Good chance for 2% increase within 2 hours

73. Limitations of the use of rules in specifying predictive event patterns Limitations: 1. Partial patterns 2. Uncertain input events 3. Complex relationship between random variables Rule = hard-coded probabilistic Relationship

74. Dynamic event prediction Time Series Prediction Graphical models Temporal Graphical models

75. Learning patterns and causalities Event Patterns Pattern and causality acquisition This is a direction to reduce the complexity of application development There are challenges in doing it – since “detected situations” are “inferred events” and may not be reflected in past events

76. 76 Security challenges of IoT Getting security feeling is a necessary condition for the success of IoT to become pervasive.

77. 77 Dangers and challenges Confusing a sensor Changing the rules of the game Abusing an actuator

78. 78 Security considerations of IoT Murder by the Internet “With so many devices being Internet connected, it makes murdering people remotely relatively simple, at least from a technical perspective. That’s horrifying,” said IID president and CTO Rod Rasmussen. “Killings can be carried out with a significantly lower chance of getting caught, much less convicted, and if human history shows us anything, if you can find a new way to kill, it will be eventually be used.” EXAMPLES: Turn off pacemakers, Shutdown car systems while driving, stop IV drip from functioning

79. 79 Confusing a sensor The same as confusing the human eyes. See things that don’t exist, don’t see things that exist, distort picture…

80. 80 Confusing a sensor Can be used to sabotage anti- crime systems, to commit fraud, or just damage something or someone…

81. 81 Confusing a sensor Example from another domain: the Twitter hoax

82. 82 Changing the rules of the game The logic is rule-based. The ease of modification can be abused to add/delete/ modify rules, change thresholds…

83. 83 Changing the rules of the game Changing data relevant for the system: maps, pictures, person’s data…

84. 84 Abusing actuators Deviating from course, shutting down, activating in wrong mode…

86. TOPIC 4 A futuristic view of the Internet of Things following Ray Kurzweil’s predictions: 86

87. 87 Driverless car Sensors that replace the human driver’s sensing, and actuators that drive the car. 2017

88. 88 Automated personal assistant Sensors that determine the context serves as active advisors. They understand your context and even listen to your conversations and give you suggestions of what to say (e.g. through google glass). 2018

89. 89 Computing implants inside the human body Sensors and actuators that fight any disease, operate in the level of cell, and reprogram the body to stop the aging process. 2020 2040 Short term: switch off our fat cells Longer term: stay young forever

90. 90 May 14, 2014

91. 91 Summary: The Internet of Everything participates in many of the predictions about the future, including Kurzweil’s singularity. The responsibility is upon us to create this future…

92. 92 My main motivation is to use the experience and knowledge I have accumulated over the years to make a better world

DEBS 2015 tutorial When Artificial Intelligence meets the Internet of Things

DEBS 2015 tutorial When Artificial Intelligence meets the Internet of Things

Recommended

More Related Content

What's hot

What's hot(20)

Viewers also liked

Viewers also liked(12)

Similar to DEBS 2015 tutorial When Artificial Intelligence meets the Internet of Things

Similar to DEBS 2015 tutorial When Artificial Intelligence meets the Internet of Things(20)

More from Opher Etzion

More from Opher Etzion(20)

Recently uploaded

Recently uploaded(20)

DEBS 2015 tutorial When Artificial Intelligence meets the Internet of Things