Research Talk at Bell Labs - IoT System Architecture and Interactions

Uploaded on

This talk discusses architecture and interaction issues for building user centred pervasive systems.

This talk discusses architecture and interaction issues for building user centred pervasive systems.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Designing Interaction for the Internet of Things Fahim Kawsar
  • 2. ‣ Senior Researcher, Ambient Media, Bell Labs (Since Nov 2010) ‣ Adjunct Researcher, Computing, Lancaster University (Since Nov 2010 -) ‣ Post-Doc Researcher, Computing, Lancaster University, UK. (Feb 2009 - Oct 2010). ‣ PhD and M.Engg. Computing, Waseda University, Japan. (April 2004 - March 2009). ‣ Microsoft Research Fellow (2006-2008).Self Introduction ‣ Monbukagakusho Scholar ( 2004-2009). ‣ Nokia Research Center (2004-2006).‣ Pervasive and Mobile Computing ‣ Smart Objects ‣ Distributed Middleware ‣ Intelligibility and UX Engineering Research Focus
  • 3. Post-PC Paradigm of Computing ‣ Computing will disappear from human perception and will operate in the periphery of human and will provide desired information service just in time understanding human context. ‣ Computing will be available everywhere and every time. Ubiquitous Computing
  • 4. InstrumentationTwo Alternatives Recreate the environment completely. Augment the existing environment and its constituents, so called Smart Environment and Smart objects that are capable of providing value added computational services. How to achieve Ubiquity
  • 5. Part - 1 Smart Objects | IoT Part - 2 Architecture Part - 3 Interaction
  • 6. “A   computa.onally   instrumented   tangible   object   with   an   established   purpose   that   augments   human   percep.on   and   is     aware   of   its   opera.onal   situa.ons   and   capable   of   providing   supplementary  services   without   compromising   its   original  appearance   and   interac.on  metaphor   significantly.”  -­‐  (Kawsar,  2007) Supplementary  Services Smart Device  Centric   Objects Situa.onal  Awareness[Beigl 2001] [Ishii, 1997] [Ambient Device] Connec.vity Perceptual  Augmenta.on[Kawsar, 2005] [Tokuda, 2004] [Intelligent Spoon, MIT] Smart Objects
  • 7. Supplementary  Services Device  Centric   Smart Situa.onal  Awareness Objects Connec.vity Perceptual  Augmenta.on Mediacup,  TeCo Music  BoNles,  MIT Smart  Furniture,  KEIO Ambient  Umbrella  Smart Objects are the Building Blocks of the Internet of Things
  • 8. Ubicomp 2005, Soc-EUSAI 2005, IE 2007, MUM 2007, SIGAPP SAC 2006
  • 9. Pervasive 2005
  • 10. ACM UIST 2007
  • 11. ACM MobileHCI 2010
  • 12. Implications
  • 13. Characteristics‣ User  Defined  Behaviour  and  Users  at  the  Center  of  Control  ‣ Evolves  Over  Time  ‣ Personalised‣  are  Endless   ‣ Web  2.0  as  an  example ‣ 2  Sided  Market  for  Diffusion  as  well  as  User  Led  Innova.on One  Smart  Object  has  mul.ple  features Mul.ple  Smart  Objects  have  same  feature
  • 14. SO/IoT System Variance SO:  Smart  Object,  F:  Func6ons,  APP:  Applica6on SO SO Stand-­‐alone  smart  objects  with   Category  1  Stand-­‐alone   F F F one  or  mul.ple Smart  Objects SO Co-­‐  smart  objects,  each   Category  2 SO SO Co-­‐   F with    one  or  mul.ple   F F F Smart  Objects APP APP APP APP APP APP Category  3   Infra-­‐structured   SO SO SO SO SO SO SO Smart  Objects F F F F F F F F F F F One  applica.on  uses  only   One  applica.on  uses   Mul.ple  applica.ons  use  mul.ple   one  smart  object  with  one  or   mul.ple  smart  objects  each  with   smart  objects  each  with  one  or   mul.ple one  or  mul.ple mul.ple ‣ Typically  applica.ons  that  run  on  smart  objects  are  context-­‐aware  applica.ons. ‣ The   structure   of   smart   object   systems   (specifically   Infra-­‐structured   and   Co-­‐  Ones)  resembles  a   typical  Distributed  System  of  mul.ple  nodes  that   are  distributed.
  • 15. Whats Next?
  • 16. Early  80’s Early  90’s Present Specific  Vendors Built  in  Apps Limited  Extensibility Numerous  Vendors Million  of  Apps All  in  One Many  Add-­‐ons,  Sensors Support  for    Extension Built  in  soZwares Thousands  of  soZwares PresentEvolu.on  of  PC  Pla_orm Evolu.on  of  Mobile  Phone ‣ Smart  objects  will  follow  the  same  trend.  Numerous  3rd  party  developers  will  build   applica.ons  for  various  smart  objects.   ‣ A  smart  object  may  have  mul.ple  features  and  mul.ple  applica.ons. ‣ Smart  objects  will  be  easily  extensible  for  value  addi.on.
  • 17. User Orientation with Do-It-Yourself (DIY) Approach Extended  by  Endusers + + Basic  Text  Chat Voice  Chat Video  Conferencing Why  Enduser  Involvement  is  Necessary  ?   ‣ Greater  User  Centric  Control ‣ Organic  Evolu.on  of  our  Living  Space ‣ BeNer  Personalisa.on  Support ‣ Frequent  Upgrade  Support ‣ BeNer  Acceptance ‣ Less  Cost ‣ DIY  Approach
  • 18. User Orientation with Do-It-Yourself (DIY) Approach Smart Mirror + + Mirror Applica6on Advanced  Feature More  Advanced  Features Basic  Display  Feature Contextual  Display  with   Extended  by   Contextual  Display Interac.on Endusers ‣ Main  Research  Challenges ‣ Smart  Object  Augmenta.on  Mechanism  with  Appropriate  System  Model ‣ Proper  Infrastructure  support  for  Applica.on  development  with  Suitable   Programming  Abstrac.ons ‣ Designing  user  interfaces  to  support  endusers  in  the  deployment,  extension   and  administra.on  processes.
  • 19. Part -2 Architecture
  • 20. ‣ Infrastructure  Design   Challenges ‣ Handling  Heterogeneity‣ Smart  Object  Design   ‣ Hiding  Implementa.on,  Access   Unifica.on,  Transparent   Challenges Communica.on ‣ Decoupling  Smart  Features ‣ Handling  Smart  Object’s  Characteris.cs ‣ Reusability ‣  Management  of  Smart   Objects  (i.e.  Discovery,  Access,  Data   ‣ Service  Unifica.on(Sensing  and   Aggrega.on,  etc.)   ‣ Suppor6ng  System  Evolu6on  (Update,   ‣  Plug  and  Play Extension  etc.) ‣    Incremental   ‣  Programming  Model  with   Deployment  and  Extension Suitable  Abstrac.ons  
  • 21. A Document Based Design Applica.on  Deployment,   Enduser Smart  Object  Deployment,   Smart  Objects  are  designed   Configura.on,  Extension Tool Configura.on,  Extension following  core-­‐cloud  model Smart  Object Applica.on F F F D E D D Smart  Object Applica.on F F D N D E Smart  Object Applica.on T F F D D Infrastructure  Independent   Infrastructure  Independent   Applica.ons Smart  Objects    F:  Smart  Feature  D:  Document  ‣ Smart  Objects  features  (as  Profiles)  are  objec.fied  by  structured  documents‣ Applica.ons  requirements   (as   Tasks)   for  smart   object   services   are  externalised  by   structured  documents‣ A   secondary   infrastructure   creates   a  spontaneous   among   the   applica.ons   and   the   smart   objects   by   mapping   applica.ons   Tasks   into   corresponding   Profiles   of   smart   objects  and  by  an  applica.on  specific  access  point.
  • 22. ‣ SODD:  Smart  Object  Descrip.on  Document Sensor   Actuator Profile  Handler Profile ‣ This  is  the  generic  file  that  represents  a   smart  object  and  is  updated  whenever  new profiles  are  added Profile  1   Profile  2   Profile  3   ‣ PDD:  Profile  Descrip.on  Document Profile  Repository Artefact No.fica.on  Module Client   ‣ Sensor  Modeling  Language  (SML)  for  Senso Memory Discovery  Module Handler   Core Type  Profile Communica.on  Module ‣ Actuator  Modeling  Language  (AML)    for   Actuator  Type  Profile Smart  Object  Wrapper‣ Wrapper  implements  the  Core-­‐Cloud  System  Model ‣ Core   acts   as   the   and   provides   basic   communica.on   and   event   management ‣ Clouds  are  plugged-­‐in  atop   the  core  as   Profiles,   One  artefact   can  have   mul.ple   profiles,  where  each  profile  is  either  sensor  or  actuator  type. ‣ Core  and  profiles  are  disseminated  as  generic  binaries
  • 23. Programming  Model ‣ The  core  of  the  Smart  Object  Wrapper  is  a  generic  binary  and  act  as  the  for  the   func.onal  features  that  the  smart  object  provides. ‣ For  each  feature  of  the  smart  object  a  profile  has  to  be  implemented. ‣ A  Library  is  provided  for  the  developers  where  Profile  act  as  the  primary  abstrac.on  and   published  it  as  a  generic  binary.public class ProximityIRProfile extends Profile { protected String position,distance; public ProximityIRProfile(String path) { super(path); position=""; distance=""; new IRSensor(this); Handles the Protocol Heterogeneity } public void setSML() Sets the Profile Output in Predefined SML Syntax { this.sml.setOutput("position", this.position); this.sml.setOutput("proximity", this.distance); this.notifyAccessPoint(); }}
  • 24. Example  Documents <?xml version="1.0"?> <artefact> <name>Mirror</name> <vendor></vendor> <profiles> <profile> ! <name>Proximity</name> ! <codebase>ArtefactSpace/Mirror/ProximityProfile/ProximityIRProfile.jar</codebase> </profile> </profiles> </artefact> Smart  Object  Descrip.on  Document<?xml version="1.0"?> <actuator><profile> <identification> ... </identification>! <name>Proximity</name> <states>! <purpose>Sensing the proximity </purpose> <state>! <type>Sensor</type> <name> ... </name>! <detector> <input> ! <identification>IR Sensor</identification> <name> ... </name>! ! <referenceFrame/> <parameter>! ! <inputs/> <MIMEdatatype> ....</MIMEdatatype> <outputs> <value> </value>..! ! <output> </parameter>! ! <name>position</name> ----- More Parameter ----! ! ! <datatype>string</datatype> </input>! ! ! <value/> <output>! ! </output> <name> ... </name>! ! </outputs> </output>! ! <parameters/> </state>! </detector> </states>! <QoS-attribute/> <installation-instruction>............ </installation-instruction> <installation-instruction>............ </installation-instruction>! </actuator></profile> (a)  With  SML (b)  With  AML Profile  Descrip.on  Document
  • 25. Develop  Smart  Object  with  Core  and  Profiles As  Generic   Binaries Basic  Display  Feature Deploy  Smart  Objects   and    Document Run  and  use  the  smart   Advanced  Feature objects  in  applica.ons Contextual  Display Close  Smart  Object Add  new  Profiles to  extend As  Generic   Binary  with  Instrumenta.on Smart  Object  Life  Cycle
  • 26. ‣ Independently  built  and  disseminated  as  generic  binary.‣ The  developers  need  to  structure  the  applica.on  atomic  ac.ons  into  explicit  tasks  and   externalize  them  in  Task  Descrip6on  Document  (TDD)‣ The   access   to  the   smart   object   service   is   unified   by   following   a   RESTful  seman.cs   using   simple  HTTP/XML  regardless  of  the  smart  objects  type,  service  and  implemented  protocol.‣ To  support  applica.on  developers,  a  simple  library  that  implements  REST  is  provided  with   both   synchronous   and   synchronous   communica.on  scheme,  where   Task  is   used  primary   abstrac.on. Enumeration<Task> vector = this.taskList.elements(); while(vector.hasMoreElements()) { Task task=(Task)vector.nextElement(); if(task.getID().equalsIgnoreCase("T1") && task.getProfileFound()){ communicator.adhocCommunicate(xmlProc.generateOutgoingMessage( Constant.TASKREQUEST,task.getID()),this.apIP, this.apPort); } } Applica.on  Development
  • 27. Example  Task  Descrip.on  Document <?xml version="1.0" encoding="UTF-8"?> <application> ! <name>Smart Display Application</name> ! <purpose>Providing Personalized Infromation with Situation Awareness</purpose> ! <binaryPath>ApplicationSpace/SmartDisplay/SmartDisplayApp.jar</binaryPath> Dynamically  Injected  by  FedNet   ! <accesspointIP></accesspointIP> ! <accesspointPort>9824</accesspointPort> during  applica.on  installa.on   ! <task-list> <task> phase  to  aNach  the  Access  Point ! ! <id>T1</id> ! ! <purpose>Measuring Proximity</purpose> ! ! <required-profile-type>Sensor</required-profile-type> ! ! <profile-name>Proximity</profile-name> <communication-mode> asynchronous</communication-mode> ! ! <profile-QoS-attribute> ! ! ! ! <qos> ! ! ! ! ! <name>latency</name> ! ! ! ! ! <datatype>int</datatype> ! ! ! ! ! <measurement-unit>millisecond</measurement-unit> ! ! ! ! ! <high-threshold>70</high-threshold> ! ! ! ! ! <low-threshold>60</low-threshold> ! ! ! ! </qos> ! ! ! ! ! ! ! ! </profile-QoS-attribute> ! </task> ---------- More Tasks--------- </task-list> </application> Task  Descrip.on  Document‣ Document  Type  Defini.on  (DTD)  are  provided  for  developers.
  • 28. Smart MirrorDevelop  Applica.on   with  REST  Support As  Generic   Binaries Mirror Deploy  Applica.on  and   Applica6on  1 externalize    Document Run  and  use  available   smart  objects Mirror Applica6on  2   Close  Applica.ons Add  new  Tasks to  extend Applica.on  Life  Cycle
  • 29. FedNet  Infrastructure Access   Access   Access   Point Point Point‣ FedNet  provides  the  associa.ons   FedNet  Core among  the  applica.ons  and  the  smart   objects  by  structural  type  matching  of  the   Applica.on Smart  Object documents. Repository Repository ‣ Applica6on  Repository  hosts  all  applica.ons  and  corresponding  documents. ‣ Smart  Object  Repository  hosts  all  smart  objects  and  their  profiles  along  with  the  documents. ‣ FedNet   Core   acts   as   the   and   performs   the   structural   type   matching   to   create   the  associa.on  by   an  access   point   for   each  applica.on.  Uses   Mul.cast  DNS  for   zeo-­‐conf  bootstraping ‣ Access   Point   specific   to   each   applica.on   is   the   point   for   applica.ons     to   access   the   smart   object’s  services.
  • 30. Applica.on  Repository Task  Specifica.on  by   Documents Ai Ai 1 FedNet  Core T3 T1 T2 T1 T2 4 2 Query  Smart  Object Spawn  Access   Generate Repository  by   applica.oni=  ∑  taski 3 Subset Matching  Documents Point Applica.on Smart  Object  Repository Applica.on Applica.on  1  2  n Ar1 Ar2 P2 P1 Hook  to   Applica.on 5 P1 P2 Smart   smart-­‐objecti=  ∑  profilei Object   Smart   P1 1 P2 Object   P1 2 Smart   Smart   Smart   Smart   Object Object Object Smart  Object Object    1 2  3 Federa.on P1       3 P2 Applica.on  Specific  Access  PointREST  Based  FedNet  InfrastructureMobiquitous  2008,  UbiComp  2008,  FGCN  2008,  ISORC  2009,  Springer  2010,  Springer  Super  2010.
  • 31. Interactions
  • 32. Things + Web Internet of Things
  • 33. Things + Web + People Internet of Things
  • 34. Object CentricSpatially FixedTemporally Constrained Current IoT Interactions
  • 35. Activity CentricSpatially DistributedTemporally Dispersed People’s Interactions
  • 36. - Declarative Modelling Technique to model Activity.- Software Infrastructure to Support Task Distribution and Intra- Object Communication.- User Interface to enable Seamless Interaction.Requirements
  • 37. Situated Flow“A situated flow is a sequential model that consists of a set of actions, stitchedtogether by a plan that specifies how the actions should be performed to achievea goal under certain constrains. In other words, a flow formalizes and maps ouractivities to certain tasks to achieve a goal. It is situated and context-aware.”Activity Model
  • 38. o Micro Activity: This type of activity is not decomposable, so a flow cannot be refined on this activity. o Macro Activity: This type of activity is decomposable and contains a link to another flow. During flow association (static refinement) or execution (dynamic refinement), this activity is replaced with the linked flow’s activity or sequence of activitiesFlow Representation and Distribution
  • 39. Blood  Pressure  Monitor  Flow Glucose  Meter  Flow (  Procedures) (  Procedures) Situated  DiscoveryStep  1 Flow  Discovery Flow  Associa.on Situated    Adapta.on Step  2 Step  3 Flow  Refinement Step  4 Flow  Execu.on Situated  Interac.on Flow Driven Interaction
  • 40. RESTful Software ArchitecturePerCom  2010,  IoT  2010.
  • 41. User Interface
  • 42. [Beigl 2001] [Ishii, 1997] [Tokuda, 2004] [Ambient Orb, NabazTag, LG Intelligent Fridge] Form factor and interaction consistency need to be maintained. Design Constrains
  • 43. DeploymentAugmentation Composition IoT Interaction ShareAssociation Extension Personalization IoT Interaction- UI Space
  • 44. For managing For managing For administratingsmart objects applications smart objects and profiles and applications GUI Interaction MobiQuituos 2008
  • 45. XHard to Understand
  • 46. Conflicting Mental Model
  • 47. Missing Tangibility
  • 48. TUI Interaction UbiComp 2008
  • 49. Introduces Spatial Distribution
  • 50. Mobile GUI+TUI Interaction
  • 51. Introduces Situational Disability
  • 52. Mobile Augmented Reality UI Interaction MobileHCI 2010
  • 53. Introduces Situational Disability
  • 54. Decomposition of the Interaction Space
  • 55. 90Spatial UI Interaction with Mobile Projector MobileHCI 2010
  • 56. Introduces Fragmentation of Attention Implications
  • 57. Needs Better Hand Eye Co-ordination
  • 58. 90Spatial UI Interaction with Wearable Projector UbiComp 2011 (Hopefully)
  • 59. Introduces Fragmentation of Attention Implications
  • 60. Better support for Collaboration
  • 61. Open Research Space
  • 62. fahim.kawsar@gmail.com