This talk discusses architecture and interaction issues for building user centred pervasive systems.

1. Designing Interaction for the Internet of Things
Fahim Kawsar

2. ‣ Senior Researcher, Ambient Media, Bell Labs (Since Nov 2010)
‣ Adjunct Researcher, Computing, Lancaster University (Since Nov
2010 -)
‣ Post-Doc Researcher, Computing, Lancaster University, UK. (Feb
2009 - Oct 2010).
‣ PhD and M.Engg. Computing, Waseda University, Japan. (April 2004
- March 2009).
‣ Microsoft Research Fellow (2006-2008).
Self Introduction ‣ Monbukagakusho Scholar ( 2004-2009).
‣ Nokia Research Center (2004-2006).
‣ Pervasive and Mobile Computing
‣ Smart Objects
‣ Distributed Middleware
‣ Intelligibility and UX Engineering
Research Focus

3. Post-PC Paradigm of Computing
‣ Computing will disappear from human perception and will operate in the
periphery of human and will provide desired information service just in time
understanding human context.
‣ Computing will be available everywhere and every time.
Ubiquitous Computing

4. Instrumentation
Two Alternatives
Recreate the environment completely.
Augment the existing environment and its constituents, so called Smart
Environment and Smart objects that are capable of providing value added
computational services.
How to achieve Ubiquity

5. Part - 1
Smart Objects | IoT
Part - 2
Architecture
Part - 3
Interaction

6. “A
computa.onally
instrumented
tangible
object
with
an
established
purpose
that
augments
human
percep.on
and
is
aware
of
its
opera.onal
situa.ons
and
capable
of
providing
supplementary
services
without
compromising
its
original
appearance
and
interac.on
metaphor
significantly.”
-­‐
(Kawsar,
2007)
Supplementary
Services
Smart Device
Centric
Objects Situa.onal
Awareness
[Beigl 2001] [Ishii, 1997] [Ambient Device]
Connec.vity
Perceptual
Augmenta.on
[Kawsar, 2005] [Tokuda, 2004] [Intelligent Spoon, MIT]
Smart Objects

7. Supplementary
Services
Device
Centric
Smart Situa.onal
Awareness
Objects
Connec.vity
Perceptual
Augmenta.on
Mediacup,
TeCo Music
BoNles,
MIT Smart
Furniture,
KEIO Ambient
Umbrella
Smart Objects are the Building Blocks of the Internet of
Things

8. Ubicomp 2005, Soc-EUSAI 2005, IE 2007, MUM 2007, SIGAPP SAC 2006

9. Pervasive 2005

10. ACM UIST 2007

11. ACM MobileHCI 2010

12. Implications

13. Characteristics
‣ User
Defined
Behaviour
and
Users
at
the
Center
of
Control
‣ Evolves
Over
Time
‣ Personalised
‣ Opportuni.es
are
Endless
‣ Web
2.0
as
an
example
‣ 2
Sided
Market
for
Diffusion
as
well
as
User
Led
Innova.on
One
Smart
Object
has
mul.ple
features Mul.ple
Smart
Objects
have
same
feature

14. SO/IoT System Variance
SO:
Smart
Object,
F:
Func6ons,
APP:
Applica6on
SO SO Stand-­‐alone
smart
objects
with
Category
1
Stand-­‐alone
F F F one
or
mul.ple
func.onali.es
Smart
Objects
SO Co-­‐opera.ve
smart
objects,
each
Category
2 SO SO
Co-­‐opera.ve
F with
one
or
mul.ple
F F F func.onali.es
Smart
Objects
APP APP APP APP
APP APP
Category
3
Infra-­‐structured
SO SO SO SO SO SO SO
Smart
Objects
F F F F F F F F F F F
One
applica.on
uses
only
One
applica.on
uses
Mul.ple
applica.ons
use
mul.ple
one
smart
object
with
one
or
mul.ple
smart
objects
each
with
smart
objects
each
with
one
or
mul.ple
func.onali.es one
or
mul.ple
func.onali.es mul.ple
func.onali.es
‣ Typically
applica.ons
that
run
on
smart
objects
are
context-­‐aware
applica.ons.
‣ The
structure
of
smart
object
systems
(specifically
Infra-­‐structured
and
Co-­‐
Opera.ve
Ones)
resembles
a
typical
Distributed
System
of
mul.ple
nodes
that
are
spa.ally
distributed.

15. Whats Next?

16. Early
80’s Early
90’s Present
Specific
Vendors
Built
in
Apps
Limited
Extensibility
Numerous
Vendors
Million
of
Apps All
in
One Many
Add-­‐ons,
Sensors
Support
for
Extension Built
in
soZwares Thousands
of
soZwares
Present
Evolu.on
of
PC
Pla_orm Evolu.on
of
Mobile
Phone
‣ Smart
objects
will
follow
the
same
trend.
Numerous
3rd
party
developers
will
build
applica.ons
for
various
smart
objects.
‣ A
smart
object
may
have
mul.ple
features
and
mul.ple
applica.ons.
‣ Smart
objects
will
be
easily
extensible
for
value
addi.on.

17. User Orientation with Do-It-Yourself (DIY) Approach
Extended
by
Endusers
+ +
Basic
Text
Chat Voice
Chat Video
Conferencing
Why
Enduser
Involvement
is
Necessary
?
‣ Greater
User
Centric
Control
‣ Organic
Evolu.on
of
our
Living
Space
‣ BeNer
Personalisa.on
Support
‣ Frequent
Upgrade
Support
‣ BeNer
Acceptance
‣ Less
Cost
‣ DIY
Approach

18. User Orientation with Do-It-Yourself (DIY) Approach
Smart
Mirror
+ +
Mirror
Applica6on
Advanced
Feature More
Advanced
Features
Basic
Display
Feature Contextual
Display
with
Extended
by
Contextual
Display
Interac.on Endusers
‣ Main
Research
Challenges
‣ Smart
Object
Augmenta.on
Mechanism
with
Appropriate
System
Model
‣ Proper
Infrastructure
support
for
Applica.on
development
with
Suitable
Programming
Abstrac.ons
‣ Designing
user
interfaces
to
support
endusers
in
the
deployment,
extension
and
administra.on
processes.

19. Part -2
Architecture

20. ‣ Infrastructure
Design
Challenges
‣ Handling
Heterogeneity
‣ Smart
Object
Design
‣ Hiding
Implementa.on,
Access
Unifica.on,
Transparent
Challenges Communica.on
‣ Decoupling
Smart
Features
‣ Handling
Smart
Object’s
Characteris.cs
‣ Reusability
‣ Suppor.ng
Management
of
Smart
Objects
(i.e.
Discovery,
Access,
Data
‣ Service
Unifica.on(Sensing
and
Aggrega.on,
etc.)
Actua.ng)
‣ Suppor6ng
System
Evolu6on
(Update,
‣ Suppor.ng
Plug
and
Play
Extension
etc.)
‣ Suppor.ng
Incremental
‣ Effec.ve
Programming
Model
with
Deployment
and
Extension
Suitable
Abstrac.ons

21. A Document Based Design
Applica.on
Deployment,
Enduser Smart
Object
Deployment,
Smart
Objects
are
designed
Configura.on,
Extension Tool Configura.on,
Extension following
core-­‐cloud
model
Smart
Object
Applica.on F F F
D E D
D Smart
Object
Applica.on F F
D N D
E Smart
Object
Applica.on T F F
D D
Infrastructure
Independent
Infrastructure
Independent
Applica.ons Smart
Objects
F:
Smart
Feature
D:
Descrip.ve
Document
‣ Smart
Objects
features
(as
Profiles)
are
objec.fied
by
structured
documents
‣ Applica.ons
run.me
requirements
(as
Tasks)
for
smart
object
services
are
externalised
by
structured
documents
‣ A
secondary
infrastructure
creates
a
spontaneous
associa.ng
among
the
applica.ons
and
the
smart
objects
by
mapping
applica.ons
Tasks
into
corresponding
Profiles
of
smart
objects
and
by
crea.ng
an
applica.on
specific
access
point.

22. ‣ SODD:
Smart
Object
Descrip.on
Document
Sensor
Actuator
Profile
Handler
Profile ‣ This
is
the
generic
file
that
represents
a
smart
object
and
is
updated
whenever
new
profiles
are
added
Profile
1
Profile
2
Profile
3
‣ PDD:
Profile
Descrip.on
Document
Profile
Repository
Artefact No.fica.on
Module Client
‣ Sensor
Modeling
Language
(SML)
for
Senso
Memory Discovery
Module
Handler
Core Type
Profile
Communica.on
Module
‣ Actuator
Modeling
Language
(AML)
for
Actuator
Type
Profile
Smart
Object
Wrapper
‣ Wrapper
implements
the
Core-­‐Cloud
System
Model
‣ Core
acts
as
the
run.me
and
provides
basic
communica.on
and
event
management
facili.es
‣ Clouds
are
plugged-­‐in
atop
the
core
as
Profiles,
One
artefact
can
have
mul.ple
profiles,
where
each
profile
is
either
sensor
or
actuator
type.
‣ Core
and
profiles
are
disseminated
as
generic
binaries

23. Programming
Model
‣ The
core
of
the
Smart
Object
Wrapper
is
a
generic
binary
and
act
as
the
run.me
for
the
func.onal
features
that
the
smart
object
provides.
‣ For
each
feature
of
the
smart
object
a
profile
has
to
be
implemented.
‣ A
Library
is
provided
for
the
developers
where
Profile
act
as
the
primary
abstrac.on
and
published
it
as
a
generic
binary.
public class ProximityIRProfile extends Profile {
protected String position,distance;
public ProximityIRProfile(String path)
{
super(path);
position=""; distance="";
new IRSensor(this); Handles the Protocol Heterogeneity
}
public void setSML() Sets the Profile Output in Predefined SML Syntax
{
this.sml.setOutput("position", this.position);
this.sml.setOutput("proximity", this.distance);
this.notifyAccessPoint();
}
}

24. Example
Documents
<?xml version="1.0"?>
<artefact>
<name>Mirror</name>
<vendor></vendor>
<profiles>
<profile>
! <name>Proximity</name>
! <codebase>ArtefactSpace/Mirror/ProximityProfile/ProximityIRProfile.jar</codebase>
</profile>
</profiles>
</artefact>
Smart
Object
Descrip.on
Document
<?xml version="1.0"?> <actuator>
<profile> <identification> ... </identification>
! <name>Proximity</name> <states>
! <purpose>Sensing the proximity </purpose> <state>
! <type>Sensor</type> <name> ... </name>
! <detector> <input>
! <identification>IR Sensor</identification> <name> ... </name>
! ! <referenceFrame/> <parameter>
! ! <inputs/> <MIMEdatatype> ....</MIMEdatatype>
<outputs> <value> </value>..
! ! <output> </parameter>
! ! <name>position</name> ----- More Parameter ----
! ! ! <datatype>string</datatype> </input>
! ! ! <value/> <output>
! ! </output> <name> ... </name>
! ! </outputs> </output>
! ! <parameters/> </state>
! </detector> </states>
! <QoS-attribute/> <installation-instruction>............ </installation-instruction>
<installation-instruction>............ </installation-instruction>! </actuator>
</profile>
(a)
With
SML (b)
With
AML
Profile
Descrip.on
Document

25. Develop
Smart
Object
with
Core
and
Profiles As
Generic
Binaries
Basic
Display
Feature
Deploy
Smart
Objects
and
Document
Run
and
use
the
smart
Advanced
Feature
objects
in
applica.ons Contextual
Display
Close
Smart
Object
Add
new
Profiles
to
extend
func.onali.es
As
Generic
Binary
with
respec.ve
Instrumenta.on
Smart
Object
Life
Cycle

26. ‣ Independently
built
and
disseminated
as
generic
binary.
‣ The
developers
need
to
structure
the
applica.on
atomic
ac.ons
into
explicit
tasks
and
externalize
them
in
Task
Descrip6on
Document
(TDD)
‣ The
access
to
the
smart
object
service
is
unified
by
following
a
RESTful
seman.cs
using
simple
HTTP/XML
regardless
of
the
smart
objects
type,
service
and
implemented
protocol.
‣ To
support
applica.on
developers,
a
simple
library
that
implements
REST
is
provided
with
both
synchronous
and
synchronous
communica.on
scheme,
where
Task
is
used
primary
abstrac.on.
Enumeration<Task> vector = this.taskList.elements();
while(vector.hasMoreElements())
{
Task task=(Task)vector.nextElement();
if(task.getID().equalsIgnoreCase("T1") && task.getProfileFound()){
communicator.adhocCommunicate(xmlProc.generateOutgoingMessage(
Constant.TASKREQUEST,task.getID()),this.apIP, this.apPort);
}
}
Applica.on
Development

27. Example
Task
Descrip.on
Document
<?xml version="1.0" encoding="UTF-8"?>
<application>
! <name>Smart Display Application</name>
! <purpose>Providing Personalized Infromation with Situation Awareness</purpose>
! <binaryPath>ApplicationSpace/SmartDisplay/SmartDisplayApp.jar</binaryPath> Dynamically
Injected
by
FedNet
! <accesspointIP>10.0.1.3</accesspointIP>
! <accesspointPort>9824</accesspointPort> during
applica.on
installa.on
!
<task-list>
<task>
phase
to
aNach
the
Access
Point
! ! <id>T1</id>
! ! <purpose>Measuring Proximity</purpose>
! ! <required-profile-type>Sensor</required-profile-type>
! ! <profile-name>Proximity</profile-name>
<communication-mode> asynchronous</communication-mode>
! ! <profile-QoS-attribute>
! ! ! ! <qos>
! ! ! ! ! <name>latency</name>
! ! ! ! ! <datatype>int</datatype>
! ! ! ! ! <measurement-unit>millisecond</measurement-unit>
! ! ! ! ! <high-threshold>70</high-threshold>
! ! ! ! ! <low-threshold>60</low-threshold>
! ! ! ! </qos>
! ! ! ! ! !
! ! </profile-QoS-attribute>
! </task>
---------- More Tasks---------
</task-list>
</application>
Task
Descrip.on
Document
‣ Document
Type
Defini.on
(DTD)
are
provided
for
developers.

28. Smart
Mirror
Develop
Applica.on
with
REST
Support As
Generic
Binaries
Mirror
Deploy
Applica.on
and
Applica6on
1
externalize
Document
Run
and
use
available
smart
objects
Mirror
Applica6on
2
Close
Applica.ons
Add
new
Tasks
to
extend
func.onali.es
Applica.on
Life
Cycle

29. FedNet
Infrastructure Access
Access
Access
Point Point Point
‣ FedNet
provides
the
run.me
associa.ons
FedNet
Core
among
the
applica.ons
and
the
smart
objects
by
structural
type
matching
of
the
Applica.on Smart
Object
documents. Repository Repository
‣ Applica6on
Repository
hosts
all
applica.ons
and
corresponding
documents.
‣ Smart
Object
Repository
hosts
all
smart
objects
and
their
profiles
along
with
the
documents.
‣ FedNet
Core
acts
as
the
run.me
and
performs
the
structural
type
matching
to
create
the
run.me
associa.on
by
crea.ng
an
access
point
for
each
applica.on.
Uses
Mul.cast
DNS
for
zeo-­‐conf
bootstraping
‣ Access
Point
specific
to
each
applica.on
is
the
point
for
applica.ons
to
access
the
smart
object’s
services.

30. Applica.on
Repository Task
Specifica.on
by
Documents
Ai Ai 1 FedNet
Core
T3
T1 T2 T1 T2 4 2 Query
Smart
Object
Spawn
Access
Generate Repository
by
applica.oni=
∑
taski 3 Subset Matching
Documents
Point
Applica.on
Smart
Object
Repository
Applica.on Applica.on
1
2
n Ar1 Ar2
P2 P1
Hook
to
Applica.on 5 P1 P2
Smart
smart-­‐objecti=
∑
profilei
Object
Smart
P1 1 P2 Object
P1 2
Smart
Smart
Smart
Smart
Object Object Object
Smart
Object
Object
1 2
3
Federa.on P1
3 P2
Applica.on
Specific
Access
Point
REST
Based
FedNet
Infrastructure
Mobiquitous
2008,
UbiComp
2008,
FGCN
2008,
ISORC
2009,
Springer
Mul.media
2010,
Springer
Super
Compu.ng
2010.

31. Interactions

32. Things + Web
Internet of Things

33. Things + Web + People
Internet of Things

34. Object Centric
Spatially Fixed
Temporally Constrained
Current IoT Interactions

35. Activity Centric
Spatially Distributed
Temporally Dispersed
People’s Interactions

36. - Declarative Modelling Technique to model Activity.
- Software Infrastructure to Support Task Distribution and Intra-
Object Communication.
- User Interface to enable Seamless Interaction.
Requirements

37. Situated Flow
“A situated flow is a sequential model that consists of a set of actions, stitched
together by a plan that specifies how the actions should be performed to achieve
a goal under certain constrains. In other words, a flow formalizes and maps our
activities to certain tasks to achieve a goal. It is situated and context-aware.”
Activity Model

38. o Micro Activity: This type of activity is not
decomposable, so a flow cannot be refined on
this
activity.
o Macro Activity: This type of activity is
decomposable and contains a link to another
flow. During flow association (static
refinement) or execution (dynamic
refinement), this activity is replaced with the
linked flow’s activity or sequence of activities
Flow Representation and Distribution

39. Blood
Pressure
Monitor
Flow Glucose
Meter
Flow
(Opera.ng
Procedures) (Opera.ng
Procedures)
Situated
Discovery
Step
1 Flow
Discovery
Flow
Associa.on Situated
Adapta.on
Step
2
Step
3 Flow
Refinement
Step
4 Flow
Execu.on Situated
Interac.on
Flow Driven Interaction

40. RESTful Software Architecture
PerCom
2010,
IoT
2010.

41. User Interface

42. [Beigl 2001] [Ishii, 1997] [Tokuda, 2004]
[Ambient Orb, NabazTag,
LG Intelligent Fridge]
Form factor and interaction consistency need to be maintained.
Design Constrains

43. Deployment
Augmentation
Composition
IoT
Interaction Share
Association
Extension
Personalization
IoT Interaction- UI Space

44. For managing
For managing For administrating
smart objects
applications smart objects
and profiles
and applications
GUI Interaction
MobiQuituos 2008

45. X
Hard to Understand

46. Conflicting Mental Model

47. Missing Tangibility

48. TUI Interaction
UbiComp 2008

49. Introduces Spatial Distribution

50. Mobile GUI+TUI Interaction

51. Introduces Situational Disability

52. Mobile Augmented Reality UI Interaction
MobileHCI 2010

53. Introduces Situational Disability

54. Decomposition of the Interaction Space

55. 90
Spatial UI Interaction with Mobile Projector
MobileHCI 2010

56. Introduces Fragmentation of Attention
Implications

57. Needs Better Hand Eye Co-ordination

58. 90
Spatial UI Interaction with Wearable Projector
UbiComp 2011 (Hopefully)

59. Introduces Fragmentation of Attention
Implications

60. Better support for Collaboration

61. Open Research
Space

62. fahim.kawsar@gmail.com
http://www.fahim-kawsar.net