Research Activities in Ambient Intelligence

Politecnico di TorinoDipartimento di Automatica e Informatica e-Lite Research Group http://elite.polito.it Research activities in Ambient Intelligence Fulvio Corno – fulvio.corno@polito.it

Home Automation System (HAS)2 Dario Bonino - Home and Building 6/11/2012 Automation Systems

Home Automation System (HAS) Home automation  Automation of the home, housework or household activity.  (remote) control of:  lighting  HVAC (Heating, Ventilation and Air Conditioning)  appliances  and other systems Home automation system  An integrated system (computer-based) offering home automation functionalities  Integrates electrical devices in a house  Through a communication network  Possibly includes devices using different communication protocols 3 Dario Bonino - Home and Building Automation Systems 6/11/2012

Building Automation System (BAS)4 Dario Bonino - Home and Building 6/11/2012 Automation Systems

Building Automation System (BAS) Building automation  The advanced functionality provided by the control system of a building  E.g., security & access control, fire detection & alarms, HVAC, lighting control, air quality, smoke detection, intrusion detection, environmental control, asset location/management Building Automation System(BAS)  A computerized, intelligent network of electronic devices designed to monitor and control the mechanical, electronic, and lighting systems in a building 5 Dario Bonino - Home and Building 6/11/2012 Automation Systems

Home vs. Building Automation Building Home Automation Automation Home Automation is almost a subset of Building Automation  Most functionalities are shared Different protocols and technologies 6 Dario Bonino - Home and Building 6/11/2012 Automation Systems

Main goals Interoperability Intelligence Man-Environment interaction Accessibility 7 e-lite - AmI+DOG 26/03/2010

Technology Overview Building Automation RS-485 Home Automation8 Dario Bonino - Home and Building 6/11/2012 Automation Systems

Pot-pourri di dispositivi9 e-lite - Smart Buildings 20/04/2012

Network Technology Bus  Powerline  MyOpen  Echelon  KNX  X10  Modbus  Insteon  Echelon  Wireless  Dali  ZigBee  CAN  Z-Wave  MBus  EnOcean 10 Dario Bonino - Home and Building 6/11/2012 Automation Systems

Main goals Interoperability  Abstract high-level Intelligence (semantic) model Man-Environment  DogOnt interaction  Intelligent Multi-protocol Accessibility Gateway  DOG (Domotic OSGi Gateway)  Modular architecture and open APIs  Open Source 11 e-lite - AmI+DOG 26/03/2010

Application Area Automation  Entertainment  Real-time Control  UPnP  CAN  DLNA  KNX, MyHome, Insteon,  General purpose Echelon, Modbus, X10, ZigBee,  Bluetooth Z-Wave, EnOcean  WiFi Lighting  Dali  (all Automation) Metering  Mbus  RS-485  KNX, Modbus, Echelon, ZigBee, Z-Wave,… 12 Dario Bonino - Home and Building 6/11/2012 Automation Systems

Challenges Integration  Different electrical requirements  Different interaction modalities  Different behaviors (temporal, etc.) Interoperation  Different protocols  Different interaction modalities  Master/slave  Peer-to-peer  Etc. Modeling  Different technologies & assumptions  A single shared, common description 13 Dario Bonino - Home and Building Automation Systems 6/11/2012

Pot-pourri di applicazioni14 e-lite - Smart Buildings 20/04/2012

Intelligent Domotic Environments15 Dario Bonino - Dog 2.3 6/11/2012

System Architecture User Mobile, Web, Home Display, User Multi Touch, Accessibility, Interface User Natural language, … Interface Interface Data ERP, Web services, Stream processors, Datawarehouse analysis Device abstraction, Event Dog abstraction, State abstraction, Bundles Rules engine, … Ethernet, Wi-Fi, USB Bus-to-IP gateway GW GW Bus-to-serial gateway Domotic bus (wired, wireless) Smart D D D D D D D D Domotic devices (switches, buttons, relays, sensors, meters,Appliance …) 16 Dario Bonino - Dog 2.3 6/11/2012

In Ontology We Trust Devices and networks exposed by means of a formal, unique, representation  DogOnt (Ontology) Applications only see DogOnt-based device descriptions  Functionalities  Notifications  Commands  States  State values Internal representations and drivers must be ontology- aware, at different degrees 17 Dario Bonino - Dog 2.3 6/11/2012

E allora? Applicazione Applicazione Applicazione Applicazione D D D D D D D D D D D D18 e-lite - Smart Buildings 20/04/2012

Una lingua franca Applicazione Applicazione Applicazione Applicazione Rappresentazione neutrale D D D D D D D D D D D D19 e-lite - Smart Buildings 20/04/2012

Architetture orizzontali e aperte Applicazione Applicazione Applicazione Applicazione Elaborazione real time Regole e scenari Servizio applicativo API Funzionalità di base Scambio dati Rappresentazione neutrale Driver di interfacciamento ai protocolli D D D D D D D D D D D D20 e-lite - Smart Buildings 20/04/2012

Applicazioni consumanodati e servizi Applicazione Applicazione Applicazione Applicazione Elaborazione real time Regole e scenari Servizio applicativo API Funzionalità di base Scambio dati Rappresentazione neutrale Driver di interfacciamento ai protocolli D D D D D D D D D D D D21 e-lite - Smart Buildings 20/04/2012

Applicazioni controllano Applicazione Applicazione Applicazione Applicazione Elaborazione real time Regole e scenari Servizio applicativo API Funzionalità di base Scambio dati Rappresentazione neutrale Driver di interfacciamento ai protocolli D D D D D D D D D D D D22 e-lite - Smart Buildings 20/04/2012

DogOnt DogOnt [ISWC08] Ontology-based formal modeling of domotic environments (OWL)  Technology independent device modeling  Operation semantics for DOG [TCE08]  Interoperation semantics through relations [AMI09]  Automatic generation of inter-operation rules [TCE09] 23 e-lite - AmI+DOG 26/03/2010

Semantic Modeling (DogOnt) RDF OWL XML XSD24 e-lite - AmI+DOG 26/03/2010

Different Aspects In the same Model A Lamp is  A thing (BuildingThing)  It is controllable (Controllable)  (On/Off) (OnOffFunctionality)  It can be either Lit or not Lit (OnOffState)  It is located in a Room (isIn Room)  It is connected to a domotic plant (isA DomoticNetworkComponent) 25 e-lite - AmI+DOG 26/03/2010

Different Aspects in the same Model Building Thing IsIn / contains Building Environment Controllable House Building Apartment Plant Electric System Lamp hasState Room OnOff hasFunctionality State Control Functionality Discrete OnOff State Functionality State Functionality26 e-lite - AmI+DOG 26/03/2010

In Practice (Protégé & OWL) <owl:Class rdf:about="#SimpleLamp"> <rdfs:comment rdf:datatype="http://www.w3.org/2001/XMLSchema#string" >Simple lamp that can be just turn on or turn off</rdfs:comment> <owl:disjointWith> <owl:Class rdf:about="#DimmerLamp"/> </owl:disjointWith> <rdfs:label rdf:datatype="http://www.w3.org/2001/XMLSchema#string" >SimpleLamp</rdfs:label> <rdfs:subClassOf> <owl:Restriction> <owl:someValuesFrom rdf:resource="#QueryFunctionality"/> <owl:onProperty> <owl:ObjectProperty rdf:about="#hasFunctionality"/> </owl:onProperty> </owl:Restriction> </rdfs:subClassOf> <rdfs:subClassOf rdf:resource="#Lamp"/> </owl:Class>27 e-lite - AmI+DOG 26/03/2010

Sample Room Model in DogOnt OffNotification OnNotification hasNotification hasNotification OnOffNotification Functionality OnOffState hasFunctionalitygeneratesCmd hasState Switch hasState Lamp hasFunctionality isIn isIn OnOffState OnOffFunctionality hasCommand Sample Room hasCommand OffCommand generatesCmd OnCommand28 e-lite - AmI+DOG 26/03/2010

DOG - Domotic OSGi Gateway http://domoticdog.sourceforge.net/29 e-lite - AmI+DOG 26/03/2010

System Architecture Mobile, Web, Home Display, Multi Touch, Accessibility, User Natural language, … Interface Data ERP, Web services, Stream processors, Datawarehouse analysis Device abstraction, Event Dog abstraction, State abstraction, Bundles Rules engine, … Ethernet, Wi-Fi, USB Bus-to-IP gateway GW GW Bus-to-serial gateway Domotic bus (wired, wireless) D D D D D D D D Domotic devices (switches, buttons, relays, sensors, meters, …)30 e-lite - AmI+DOG 26/03/2010

The KNX demo box D GW31 e-lite - AmI+DOG 26/03/2010

The Z-Wave demo box GW D32 e-lite - AmI+DOG 26/03/2010

Texas Intruments eZ430-Chronos33 e-lite - AmI+DOG 26/03/2010

DogEye user interface34 e-lite - AmI+DOG 26/03/2010

DogEye Video35 e-lite - AmI+DOG 26/03/2010

DogEye with an eye tracker36 e-lite - AmI+DOG 26/03/2010

Mobile House Control37 e-lite - AmI+DOG 26/03/2010

System Architecture Mobile, Web, Home Display, Multi Touch, Accessibility, User Natural language, … Interface Data ERP, Web services, Stream processors, Datawarehouse analysis Device abstraction, Event Dog abstraction, State abstraction, Bundles Rules engine, … Ethernet, Wi-Fi, USB Bus-to-IP gateway GW GW Bus-to-serial gateway Domotic bus (wired, wireless) D D D D D D D D Domotic devices (switches, buttons, relays, sensors, meters, …)38 e-lite - AmI+DOG 26/03/2010

OSGi Alliance Open Services Gateway Initiative  defines open specifications for  Creating and delivering distributed and concurrent applications, called bundles  Bundle deployment and installation (core specification)  Communication among bundles (core specification)  Devices and drivers management (service compendium specification)39 e-lite - AmI+DOG 26/03/2010

OSGi definitions  OSGi frameworks  Implementation in JAVA of the OSGi specifications (equinox, oscar, ..)  Manage the life cycle of the bundles  Resolve the dependences among bundles  Manage Service Registry  Bundles  Jar file  contain java classes and resources  Provide and use services  Services  java objects registered in the framework  Defined by java interfaces40 e-lite - AmI+DOG 26/03/2010

High-Level Architecture OSGi - based 2 main layers 3 main bundle groups  Core  Drivers  Add-ons 41 Dario Bonino - Dog 2.3 6/11/2012

Core DogXMLEndPoint DogRESTEndPoint DogStateMonitor DogScheduler DogExecutor DogDeviceManager DogNotificationManagerDogAutoStart DogDeviceFactory DogSimpleHouseModel DogSemanticHouseModel DogOntLibrary DogDeviceModel DogConfigurator DogLogger Dog2Library DogJaxBLibrary DogSemanticLibrary MeasureLibrary org.rxtx 42 Dario Bonino - Dog 2.3 6/11/2012

Command handling (1) Application Command Dog43 Dario Bonino - Dog 2.3 6/11/2012

Command handling (2) Command DogXMLEndPoint DogRESTEndPoint DogStateMonitor DogScheduler DogExecutor DogDeviceManager DogNotificationManagerDogAutoStart DogDeviceFactory DogSimpleHouseModel DogSemanticHouseModel DogOntLibrary DogDeviceModel DogConfigurator DogLogger Dog2Library DogJaxBLibrary DogSemanticLibrary MeasureLibrary org.rxtx KNX Modbus Echelon OpenWebNet ZWave Network Message 44 Dario Bonino - Dog 2.3 6/11/2012

Notification handling (1) Application Notification Dog45 Dario Bonino - Dog 2.3 6/11/2012

Notification handling DogXMLEndPoint DogRESTEndPoint Rules DogState Dog Dog DogDevice DogNotification Monitor Scheduler Executor Manager ManagerDogAutoStart DogDeviceFactory DogSimpleHouseModel PowerBundle DogSemanticHouseModel DogOnt DogDevice PowerModel Library Model Bundle DogConfigurator DogLogger DogSemantic Dog2Library DogJaxBLibrary MeasureLibrary org.rxtx Library KNX Modbus Echelon OpenWebNet ZWave 46 Dario Bonino - Dog 2.3 6/11/2012

In DogOnt we trust (3) Thing Configuration Lamp1 DogOnt Building thing Controls Instances Controllable UnControllablle Button1Appliances HousePlants Lamp2 Controls Auto Brown Electrical HVAC Generation White Security Goods Goods System System Power Consumption Device Semantic Representations House Applications Model Eletric Power Consumption Configuration Power Lamp1Pw = 18W Model 47 Dario Bonino - Dog 2.3 6/11/2012

Device Category (structure) Lighting public interface Lamp extends Lighting { isA public static int MATCH_TYPE=100; public static int MATCH_SUB_TYPE=50; public static int MATCH_MANUFACTURER=0; Lamp OnOffState public static String ON_OFF_STATE = "OnOffState"; hasState instance public void off(); public void on(); } Lamp hasFunctionality OnOffFunctionality hasCommand hasCommand OnCommand OffCommand48 e-lite - AmI+DOG 26/03/2010

Device Model (Structure) public class DogLamp extends DogDevice implements Lamp { public DogLamp(BundleContext context, Properties properties){ super(context); if(properties==null) properties=new Properties(); properties.put(Constants.DEVICE_CATEGORY, Lamp.class.getCanonicalName()); this.setDeviceProperties(properties); this.states.put(Lamp.ON_OFF_STATE, new DogStatusVariable( properties.getProperty(DogDeviceCostants.DEVICEURI), new StatusVariable(Lamp.ON_OFF_STATE, StatusVariable.CM_DER, "-"), "current state of Lamp",true)); this.registerDevice(Lamp.class.getName()); this.registerStatusVariable(); } @Override public void off(){ OnOffState Lighting ((Lamp) this.driver).off(); } @Override hasState public void on(){ isA ((Lamp) this.driver).on(); } } Lamp Lamp hasFunctionality OffCommand instance hasCommand OnOffFunctionality OnCommand49 e-lite - AmI+DOG 26/03/2010

Device Category (Hi-Fi)package it.polito.elite.domotics.model.devicecategory; /*** HiFiCategory - automatically generated by DogOnt2Dog*/public interface HiFi extends Entertainment{ public static int MATCH_TYPE=100; public static int MATCH_SUB_TYPE=50; public static int MATCH_MANUFACTURER=0; public static String STAND_BY_ON_OFF_STATE = "StandByOnOffState"; public static String SOURCE_SELECTION_STATE = "SourceSelectionState"; public static String PLAY_STATE = "PlayState"; public static String TUNER_STATE = "TunerState"; public static String VOLUME_LEVEL_STATE = "VolumeLevelState"; public void stop(); public void stepUp(); public void rew(); public void set(Object value); public void setVolume(Integer volume); public void next(); … public void goToTrack(Integer trackNumber);}50 e-lite - AmI+DOG 26/03/2010

Automatic Generation Template based  Common structure of device categories and device models Fill “placeholders” (possibly empty) with information extracted from DogOnt  SPARQL Works on the schema only 51 e-lite - AmI+DOG 26/03/2010

Automatic Generation (recipe) Device Categories Device Models 1. GetAllDevicesWithAncestors 1. GetAllDevicesWithAncestors 2. Foreach device 2. Foreach device 1. getDeviceCommands 1. getDeviceCommands 2. getDeviceStates 2. getDeviceStates 3. Write the device category 3. Write the device model code code 3. Compile all 3. Compile all 4. Jar all with manifest 4. Jar all with manifest (bundleize) (bundleize) 5. Use in Dog2.0 5. Use in Dog2.052 e-lite - AmI+DOG 26/03/2010

Queries (1) - Devices Get all devices with ancestors  SELECT DISTINCT ?x WHERE { ?x rdfs:subClassOf dogont:Controllable FILTER (?x!=owl:Nothing)} ORDER BY ?x  Recurse over ?x and call  OntClass deviceClass = this.modelLoader.getPlainOntModel().getOntCla ss(deviceURI);  ExtendedIterator iter = deviceClass.listSuperClasses(); 53 e-lite - AmI+DOG 26/03/2010

Queries (2) - Commands Non-Parametric Commands Parametric Commands SELECT DISTINCT ?commandValue WHERE { SELECT DISTINCT ?commandValue ?commandParamValue WHERE { dogont:"+deviceClass+" rdfs:subClassOf dogont:"+deviceClass+" rdfs:subClassOf [rdf:type [rdf:type owl:Restriction; owl:onProperty owl:Restriction; owl:onProperty dogont:hasFunctionality; dogont:hasFunctionality; owl:someValuesFrom owl:someValuesFrom ?functionality] . ?functionality ?functionality] . ?functionality rdfs:subClassOf dogont:ControlFunctionality . ?functionality rdfs:subClassOf [rdf:type owl:Restriction; owl:onProperty rdfs:subClassOf dogont:ControlFunctionality. dogont:hasCommand; owl:someValuesFrom ?command] . ?command ?functionality rdfs:subClassOf [rdf:type rdfs:subClassOf dogont:ParametricCommand . ?command owl:Restriction; owl:onProperty rdfs:subClassOf [rdf:type owl:Restriction; owl:onProperty dogont:hasCommand; owl:someValuesFrom dogont:realCommandName; owl:hasValue ?commandValue] . ?command] . ?command rdfs:subClassOf ?command rdfs:subClassOf [rdf:type owl:Restriction; dogont:NonParametricCommand . ?command owl:onProperty dogont:commandParamName; owl:hasValue rdfs:subClassOf [rdf:type owl:Restriction; ?commandParamValue]} ORDER BY ?commandValue owl:onProperty dogont:realCommandName; owl:hasValue ?commandValue]} ORDER BY ?commandValue54 e-lite - AmI+DOG 26/03/2010

Queries(3) - States SELECT ?state WHERE{ dogont:"+deviceClass+" rdfs:subClassOf [rdf:type owl:Restriction; owl:onProperty dogont:hasState; owl:someValuesFrom ?state] } ORDER BY ?state 55 e-lite - AmI+DOG 26/03/2010

Accessibility / Disability56 e-lite - AmI+DOG 26/03/2010

Testing Dog in a Real Home57 e-lite - AmI+DOG 26/03/2010

http://elite.polito.it/spchains Stream Processing spChains spChains Stream Pattern Match / Alerts Processing Block Data-centric Event Sources Application Event Drains Field Stream Aggregate / Computed (e.g., E.R.P.) Data Streams Processing Measures ChainsField-level sensors / data sources Chain Definition Industrial Stream processing engine Stakeholders (e.g., Management) 58 e-lite - Smart Buildings 20/04/2012

Alcuni esempi di spChains Ogni spChain è definita in XML Libreria di spBlock parametrici (13++) Creazione automatica di query sul motore Esper (o altri) Open Source 59 e-lite - Smart Buildings 20/04/2012

60 e-lite - Smart Buildings 20/04/2012

Visual rule builder61 e-lite - AmI+DOG 26/03/2010

Visual Rules Builder: video62 e-lite - AmI+DOG 26/03/2010

SimulationStateCharts Apache Commons SPARQL 63 e-lite - AmI+DOG 26/03/2010

Simulation OffStateValue OnStateValue Lamp Machine Template Lamp1 Lamp1hasStateValue hasStateValue OnOffStateLamp1 &id;OffState &id;lampOn hasState Lamp isA SampleLamp1 &id;lampOff &id;OnState hasFunctionality OnOffFunctionality hasCommand hasCommand OffCommand OnCommand64 e-lite - AmI+DOG 26/03/2010

Simulation – Bank Door65 e-lite - AmI+DOG 26/03/2010

Intelligence + Security OWL Drools SWRL SPARQL66 e-lite - AmI+DOG 26/03/2010

Rules Generation (DOG1.0) Query Rule SELECT DISTINCT ?x ?y ?v ?c ?d when ?class ?cv WHERE { ?x a dev:Device( id == "http://elite.polito.it/ontologies/simple dogont:Controllable . ?y a home.owl#"?x") dogont:Controllable . ?x cmd:StateCommand(state =="?v") dogont:controlledObject ?y . ?x status:DeviceStatus( status == cmd, dogont:hasFunctionality ?f . ?f device == dev) dogont:hasNotification ?n . ?n rulesCore:RulesCore() dogont:notificationValue ?v . ?n message: DogMessage(dataType == dogont:generateCommand ?c . ?d DogBundle.DataTypeEnum.DEVICE_STATUS, data == status) dogont:hasFunctionality ?f2 . ?f2 then dogont:hasCommand ?c . ?c Device newDev = new rdf:type ?class . ?class Device("http://elite.polito.it/ontologies rdfs:subClassOf /DOGHouseModel.owl#"+?y+"); dogont:DiscreteCommand . ?class DeviceStatus newStatus = new DeviceStatus(newDev, new rdfs:subClassOf [rdf:type StateCommand("+?cv+”,StatusTypeEnum.DISCR owl:Restriction; owl:onProperty ETE)); dogont:realCommandName; DogMessage myCmd = new owl:hasValue ?cv]} DogMessage(MessageTypeEnum.CMD,new GregorianCalendar(), message.getDataType(), newStatus); rulesCore.takeActions(myCmd);67 e-lite - AmI+DOG 26/03/2010

Energy management68 e-lite - AmI+DOG 26/03/2010

In-Home Display videosDFP: Direct Feedback of GSE: Goal Setting foristantaneous Power consumed Energy 69 e-lite - AmI+DOG 26/03/2010

Energy Awareness70 e-lite - AmI+DOG 26/03/2010

Contacts Web site:  http://elite.polito.it Dog2.0 information, documentation and downloads  http://domoticdog.sourceforge.net Publications:  http://elite.polito.it/publications-mainmenu-81 Contact e-mail:  fulvio.corno@polito.it 71 e-lite - AmI+DOG 26/03/2010

Get involved! User interfaces / User experience Artificial intelligence / Semantic modeling Wireless [sensor] networks Ambient Assisted Living applications Stream data processing Extension to Buildings and Industrial settings Energy savings / Energy management Linked-data information exchange Device modeling / Environment modeling … 72 e-lite - AmI+DOG 26/03/2010

Licenza d’uso Questa presentazione è rilasciate con la licenza Creative Commons “Attribuzione-Non commerciale-Condividi allo stesso modo 2.5 Italia” Siete liberi di riprodurre e modificare quest’opera, per scopi non commerciali, e citando la fonte. Eventuali versioni modificate dovranno essere rilasciate con la stessa licenza Testo completo della licenza:  http://creativecommons.org/licenses/by-nc-sa/2.5/it/ 73 e-lite - AmI+DOG 26/03/2010

Research Activities in Ambient Intelligence

by Fulvio Corno

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 1

Statistics