Lecture at ICTAC School 2021: 18th International Colloquium on Theoretical Aspects of Computing, Nazarbayev University, Nur-Sultan, Kazakhstan, 1st September 2021. https://www.alandix.com/academic/talks/ICTCS-School-2021/ The second part focuses on a specific area, the formal specification and analysis of systems that have both physical and digital aspects. This will incude using physigrams, a extension of finite state networks for describing interactions with physical devices such hand-held controllers. We will also look at the ways formal analuysis contributed to the design of a internet-enabled ‘cafe open’ sign – IoT in action!

1. Modelling interactions:
digital and physical
Alan Dix
https://www.alandix.com/academic/talks/ICTCS-School-2021/


ICTAC 2021

2. 
Part II – getting physical
physigrams
sensor-rich interactions and IoT


ICTAC 2021

3. modelling physical things
have a go …
– what was hard or easy?
– did you need to include the environment?
physigrams
– modelling the device ‘unplugged’
– connecting digital and physical
IoT shop open sign
– modelling human processes
… including uncertainty

5. multiple feedback loops physical–logical
mappings
physical aspects
knobs, dials,
buttons, location,
orientation
virtual aspects
screens,
lights,
buzzers, speakers
(ii) physical effects
(iii) virtual effects
show message,
turn light on
motors, effectors
(a) physical manipulation (i) sensed inputs
logical
system
A B
C
(c) felt feedback
(d)‘electronic’feedback
(b) perceived state
see message on screen
resistance,
? physical sounds ?
turn knob, press button
effects on
logical objects
device
physical
environment
D
(iv) physical effects
controlling
external things
(e) physical feedback
notice light is on,
kettle boils
semantic
feedback

6. multiple feedback loops
physical aspects
knobs, dials,
buttons, location,
orientation
virtual aspects
screens,
lights,
buzzers, speakers
A
(c) felt feedback
(b) perceived state
(a) physical manipulation
turn knob, press button
device

7. multiple feedback loops
physical aspects
knobs, dials,
buttons, location,
orientation
virtual aspects
screens,
lights,
buzzers, speakers
logical
system
A
(ii) physical effects
motors, effectors
(i) sensed inputs
B
(c) felt feedback
physical–logical
mappings
(b) perceived state
(a) physical manipulation
turn knob, press button
device

8. multiple feedback loops physical–logical
mappings
physical aspects
knobs, dials,
buttons, location,
orientation
virtual aspects
screens,
lights,
buzzers, speakers
(ii) physical effects
motors, effectors
(a) physical manipulation (i) sensed inputs
logical
system
A B
(iii) virtual effects
show message,
turn light on
C
(c) felt feedback
(b) perceived state
(d)‘electronic’feedback
see message on screen
resistance,
? physical sounds ?
turn knob, press button
device

9. multiple feedback loops physical–logical
mappings
physical aspects
knobs, dials,
buttons, location,
orientation
virtual aspects
screens,
lights,
buzzers, speakers
(ii) physical effects
(iii) virtual effects
show message,
turn light on
motors, effectors
(a) physical manipulation (i) sensed inputs
logical
system
A B
C
(c) felt feedback
(d)‘electronic’feedback
(b) perceived state
see message on screen
resistance,
? physical sounds ?
turn knob, press button
effects on
logical objects
device
physical
environment
D
(iv) physical effects
controlling
external things
(e) physical feedback
notice light is on,
kettle boils

10. multiple feedback loops
the GUI fallacy … semantic feedback is NOT enough
physical–logical
mappings
physical aspects
virtual aspects
screens,
lights,
buzzers, speakers
(ii) physical effects
(iii) virtual effects
show message,
turn light on
motors, effectors
(a) physical manipulation (i) sensed inputs
logical
system
A B
C
(c) felt feedback
(d)‘electronic’feedback
(b) perceived state
see message on screen
resistance,
? physical sounds ?
turn knob, press button
effects on
logical objects
device
physical
environment
D
(iv) physical effects
controlling
external things
(e) physical feedback
notice light is on,
kettle boils
semantic
feedback
knobs, dials,
buttons, location,
orientation

11. multiple feedback loops physical–logical
mappings
physical aspects
knobs, dials,
buttons, location,
orientation
virtual aspects
screens,
lights,
buzzers, speakers
(ii) physical effects
(iii) virtual effects
show message,
turn light on
motors, effectors
(i) sensed inputs
logical
system
B
C
(d)‘electronic’feedback
see message on screen
resistance,
? physical sounds ?
(a) physical manipulation
A
(c) felt feedback
(b) perceived state
turn knob, press button
effects on
logical objects
device
physical
environment
D
(iv) physical effects
controlling
external things
(e) physical feedback
notice light is on,
kettle boils
semantic
feedback
the GUI fallacy … semantic feedback is NOT enough

13. model physical device states
the device ‘unplugged’
the device ‘unplugged’
switch
UP
DOWN
user pushes
switch up
and down
two visible …
and feelable …
states
physigram

14. model logical system
light
OFF
ON
two states
of the system

15. physical–logical mapping
light
OFF
ON
switch
UP
DOWN
user pushes
switch up
and down

16. exposed state
particularly easy to understand and use
but not always 1–1
light
OFF
ON
switch
UP
DOWN
user pushes
switch up
and down
mapping 1–1

17. controlled state

18. compliant interaction
(1) system state visible through control
(2) system and user have similar effects
press
down
UP
DOWN
press
up
kettle switch
system
down
system
down
BOILING
Temp
< 100
POWER
OFF
POWER
ON
system state

19. 19
bounce-back button
OUT
IN
user pushes
switch in
bounce
back
computer
OFF
ON
(a) (a)
switch
(a)
(z)

20. 20
initial pressure on exposed state switch
UP
DOWN
PART
DOWN
PART
UP
press
down
press
down
press
up
press
up
switch
‘gives’
switch
‘gives’
press
down
UP
DOWN
press
up
shorthand

21. time-dependent devices (continued pressure)
21
CENTRE
IN
LEFT
IN
RIGHT
IN
twist
left
twist
right
CENTRE
OUT
LEFT
OUT
RIGHT
OUT
twist
left
twist
right
pull out
pull out
pull out
minidisk

22. 22
CENTRE
LEFT
RIGHT
twist left
(down)
twist right
(up)
1
down
minidisk knob
2
3
 (up)
 (down)
track selected
17
up
up
up
up
down
down
down

23. give it to designer

25. Tiree projects

26. Tiree Tech Wave

27. island projects – slow research
Frasan - mobile heritage app
OnSupply – renewable energy awareness
Projected touch-table
TireeConnect – island communication
gossip is not enough!
TireeDashboard
General pattern
understand – act – reflect
… takes time

28. the chip van that tweets …
… and the internet connected Open sign

29. prepare
to open
café
open
tidying
up
café
empty
arrive
at café
open café
door
close café
doors
go home
café
open

30. digital intervention – real IoT!

31. tidying
up
prepare
to open
prepare
to open
open sign off
open sign on
café
open
café
open
tidying
up
café
empty
(i) arrive
at café
café
empty
(a) forget
sign
(iii) open
café doors
(ii) switch
on sign
(b) remember
sign
(iv) switch
off sign
(v) close
café doors
(d) remember
sign
(c) forget
sign
(vi) go home

32. reliability
of sensor
definite yes
definite no
desirability of intervention
definite not maybe definite yes
uncertain
✔
✗ ✔
✗
? ?

33. tidying
up
prepare
to open
prepare
to open
open sign off
open sign on
café
open
café
open
tidying
up
café
empty
(i) arrive
at café
café
empty
(a) forget
sign
(iii) open
café doors
(ii) switch
on sign
(b) remember
sign
(iv) switch
off sign
(v) close
café doors
(d) remember
sign
(c) forget
sign
(vi) go home

35. takeaways – physical–digital systems
you can model them
models need to encompass
– physical interactions
– digital interactions
– aspects of external context (physical and social)
embrace and understand uncertainty