Matthew is a professor in the School of Computing Science at the University of Glasgow. His work focuses on data visualisation and analytics, data ethics and ethical systems design, and mobile and ubiquitous computing. Matthew worked in industrial research labs, including Xerox PARC in Palo Alto, California, before returning to Scotland in 2000. Since then he’s been an academic at University of Glasgow, leading projects exploring topics such as mobile computing for health and fitness, user experience design that bridges digital and analogue media, using commercial app stores for user trials, and large scale data analytics and visualisation. Today he’ll be talking about an approach to the design of complex systems that could perhaps be better known outside of the world of research: ’seamful design’, that started at PARC in the 1980s, and which he and his research group have advanced over the past years.

1. Seamful Design
Matthew Chalmers
School of Computing Science

2. Seamless or seamful
No system can always be invisible and seamless, and
yet too often we design as if they are
User studies (and philosophy!) detail how technologies appear,
disappear and reappear, in the course of interaction
People build understanding through interaction with each other
and with systems, over time
Sometimes the system/use has to be visible and
analytic
Considering how to control it, so that it works better for you
Learning about the system
Teaching others how to use it
Considering how to present yourself to others through it
Working on or adapting the system itself
Handling breakdowns, gaps and seams in the system

3. Seams and Seamful Design
Seams inevitably show through in interaction
Gaps and ‘losses in translation’ in complex media, models, infrastructures…
One can smooth such seams out, ignore them… or ‘seamfully’ design to let
people accommodate or even take advantage of them
Seamful design involves selecting seams to reveal as interaction
resources
Design focus on where complex models/infrastructures show through
Seamful games as ‘labs’ for design ideas
Treasure: a wi-fi network’s coverage, and how it breaks, as a design resource
NetBreak: a neural network’s classification, and how it breaks, as a design
resource

4. Seams and Seamful Design
Seams inevitably show through in interaction
Gaps and ‘losses in translation’ in complex media, models, infrastructures…
One can smooth such seams out, ignore them… or ‘seamfully’ design to let
people accommodate or even take advantage of them
Seamful design involves selecting seams to reveal as interaction
resources
Design focus on where complex models/infrastructures show through
Seamful games as ‘labs’ for design ideas
Treasure: a wi-fi network’s coverage, and how it breaks, as a design resource
NetBreak: a neural network’s classification, and how it breaks, as a design
resource

5. Treasure (2004)
Two teams of players, Blue and Green.
Each player has PDA+wi-fi+GPS. Last
known positions shown on map in app
Some game interactions take
advantage on being in the wi-fi network,
some take advantage on being out of it
Picking up ‘coins’, stealing them from others
(‘pickpocket’), uploading them safely to gain
points, turning on a shield to block thieves
System design reveals where and how
people have used network and its
gaps… to help people play now

6. Replaying a Game of Treasure
Study via people
seeing video and maps
of what they did,
recounting what they
did, swapping stories,
showing off…
The system, its use,
and other forms of
interaction around use
are interdependent

7. NetBreak (2019)
Our game design was inspired by
the success of one pixel attacks
on deep networks for image
classification, facial recognition,
etc.
Such systems for classification
and recognition are an
increasingly pervasive part of data
infrastructures
Difficult to design well, highly sensitive
to the data fed to them, and… not so
hard to break
Single player game app reveals
how an image-based ‘deep
learning’ network succeeds/fails to

8. NetBreak: a seamful game for ‘deep learning’ neural
networks
In each game round, you start with an image and its classification, and then can attack it
by changing a limited number of pixels in a limited amount of time. Gain a point each
time you make the network misclassify the image.

9. NetBreak: a seamful game for ‘deep learning’ neural networks
As players advance in their play, and discuss their play, they build up a
better understanding of how the network actually classifies images

10. It’s not all fun and games
How might this transfer to… health data, for example?
Caldicott: Citizens to be given more ownership of their personal (inc. medical)
data
How to support transparency of data processing, as well as agency and
negotiability?
Problem of the hidden architectures of data, and the inferences based on
such data
What if a person’s phone stored all of their personal data?
Playful designs might not be appropriate (always), but seamful ideas might
still help
Give the citizen more control over data sharing via their phone, apps, etc.
Supplement this with history that lets people understand how to apply such
control

11. Sensing and modelling are advancing quickly
Depressive episodes via GPS (Canzian and Musolesi)
Schizophrenia via mobility, conversation, activity, smartphone usage, the
ambient acoustic environment (Wang et al.)
Social anxiety (Boukhcheba et al.) via GPS & phone/SMS stats
Memory, executive function, language, intelligence, etc. via taps, scrolls,
clicks, etc. (Mindstrong)
Also: uni students’ GPA predicted (±0.18), via phone data (StudentLife)
And then, of course, there is Facebook, Cambridge Analytica, and all that…
and predicting personality types, political views, sexual preferences, etc. via
media use

12. Using history for dynamic consent
Redesigning a system for predicting social anxiety
via stats of GPS/phone/messaging data
Initial analysis can be done on phone, so we see no
personal data
Additional detailed ‘raw’ data can be sent to our server, to
support experiments in new predictive models
App offers control over data sensing, based on
data use
Tools to change sensor settings, level of battery drain…
History of what data was sent where
Messages back about who has used it for what (e.g.
prediction quality for particular sensor setting and analytic
method)

13. Using history for dynamic consent
Redesigning a system for predicting social anxiety
via stats of GPS/phone/messaging data
Initial analysis can be done on phone, so we see no
personal data
Additional detailed ‘raw’ data can be sent to our server, to
support experiments in new predictive models
App offers control over data sensing, based on
data use
Tools to change sensor settings, level of battery drain…
History of what data was sent where
Messages back about who has used it for what (e.g.
prediction quality for particular sensor setting and analytic
method)

14. Conclusion
Seamful design
Select appropriate seams to reveal as a resource for user interaction
Weave in histories of use — not just one time, state, location or stage
Support users’ development of understanding, accommodation and appropriation
Offers ideas applicable to many forms of information, infrastructure &
interaction
Better support for users’ understanding and agency is increasingly important
Design for legibility of ongoing operation, and of histories of use/effects of that data
Opening up new designs that dynamically sustain informed consent, GDPR
compliance, etc.

15. Thanks
matthew.chalmers@glasgow.ac.uk
www.matthewchalmers.net