It’s 2013. The shift to mobile is well and truly upon us, we’re at the transition point where Mobile Internet overtakes Desktop Internet usage and there is no going back. If you’re not designing responsively now then you better get cracking because what comes next is a big change to the way we design and build web experiences for humans. The shift to mobile started making us think about devices, however almost all of our discussion is around what devices we support and where do we set our breakpoints. Could this focus on device capability be masking something bigger happening in the way humans are starting to behave with the web? What if the fragmentation we’re seeing on Android is merely a glimpse into the device fragmentation of the future? What happens when a users’ experience can range from the interface of a watch to that of a building? How do we communicate with someone when they are walking down the street trying to locate something compared to kicking back on the sofa? How do we even know when they are doing one and not the other? Contextually Responsive Design is quickly going to become a necessity - but this isn’t personalisation 2.0 or Content First under a different name. To truly design engaging experiences we need to consider how context shapes our behaviour. As Web Designers and Developers we’ve traditionally worked with people “sitting down to compute”. What happens when someone no longer wants to sit down at a computer to do something; ever? This talk will start from the point of late 2013 and look forward six years. How do we expect people to behave? How will we design systems to cope, and what are the contexts in which people will use the web during its next age? Given at Melbourne Be Responsive on 10 September 2013.

1. Designing a
Moving Experience
Andrew Fisher @ajfisher
Be Responsive, 10 September 2013
Hi! My name is Andrew Fisher and I’m an interaction
researcher. Tonight I’m going to talk to you about how
to design a moving experience for the next 25 minutes.

2. 8 years difference
I imagine everyone has seen this picture by now. In this 8
years between popes it shows a striking technological
change. However what’s interesting for me is that this 8
year period depicts a fundamental change in behaviour
too.

3. 2005
Reuters / Jerry Lampen
This pic shows it better. Look down here at what people
are holding...
Cameras you need to LOOK through to see what you’re
shooting.
image: http://www.businessinsider.com.au/vatican-square-2005-and-2013-2013-3

4. 2013
AP
Here, there are more cameras because there are more phones. And the one person
with an iPad.
But we’ve moved from look through to look at because now EVERYTHING has a
screen.
In the span of 8 years, the behaviour of hundreds of millions of people has changed
irreversibly after being culturally ingrained for about a hundred years.
These pics show how behaviour is changing and is changed by new interaction types
afforded by our devices.
image: http://www.businessinsider.com.au/vatican-square-2005-and-2013-2013-3

5. Time at human scale
Flickr (cc): Pink Sherbet Photography
So apparently the average span of a pope is about six and a half years.
As a timespan it’s nice to work with as it doesn’t feel as long as a decade. It’s the
division of time we use when we grow up that covers pre school, junior school and
senior school. It’s a very human time scale and it’s also long enough to see changes
happen.
image (cc): http://www.flickr.com/photos/pinksherbet/179279964/

6. Moore’s Law
7 years also represents about a 10x increase in
electronic capability according to Moore’s law.
So this time period is a very interesting one to observe
for many reasons.

7. Not many years ago...
flickr (cc) Zitona
6-7 years ago we didn’t have iPhones or Android phones.
We barely had voice recognition at all, let alone on our phones.
The nintendo Wii was just showing us a future where waving your arms around at a
machine wasn’t a sign of madness but something quite normal people would be
happy to do.
But I feel we’ve gone off track. Responsive design has focussed too much on our
devices and on technical solutions and too little on the behaviour that goes
alongside.
image: (cc) http://www.flickr.com/photos/zitona/2098321266/

8. Design for humans
flickr (cc) jonrawlinson
So let’s look forward and see what the next 6 years will
bring us and how we will design moving experiences in
the future.
Tonight I want you to think about how responsive design
goes beyond just our technology. About how we design
for humans and their behaviour not the devices
themselves so that experiences are more contextually
aware.
To do that we are going to cover:
image (cc): http://www.flickr.com/photos/london/106913032

9. Agenda
1. New device types
2. New interaction methods
3. Computing as substrate
How new device types will influence our design.
What new methods of interaction should we consider
And how computing will become embedded into our
environment.

10. New devices
flickr (cc) x-ray delta one
Lets start with new devices coming up over the next
handful of years. What will they do? How will they change
our behaviour?
image (cc): http://www.flickr.com/photos/x-ray_delta_one/4870722078

11. Glass-like
flickr (cc) guiseppe.costantino
Google is betting the future on Glass and there’s a lot of
noise about this right now because it taps into an
Augmented Reality meme going back at least as far as
terminator.
image (cc): http://www.flickr.com/photos/69730904@N03/8813585832

12. Niche applications
flickr (cc) loiclemeur
There will be some very good, although highly niche use
cases for glass like devices. Though I can’t see them
being used all the time as they are intrusive and likely to
be psychologically draining.
image (cc): http://www.flickr.com/photos/loiclemeur/
8699901706/

13. Retail applications
flickr (cc) Zemlinki!
Mach 4 X-Trainer
$189.99
★★★★☆
Other prices:
$149.99 (Amazon)
$134.56 (Shoes)
I can see good use cases in retail, where you pick up a
product and shows us competitive pricing, features and
customer ratings.
There’s some domain specific benefits here but they
mostly already work on mobile however they give you
the benefit of efficiency because you hands become
more free.
In this time span there’s likely to be a lot of hype but a
lot of disillusionment as well.
image (cc): http://www.flickr.com/photos/zemlinki/196284587/

14. Microscreens
flickr (cc) Ivva!
Next up are microscreens and in particular smart watches. These are much closer to
reality and there are plenty of companies playing here right now with products or
prototypes.
Smart watches lay in an interesting area of ambient technology at the intersection of
relatively high information density and a high degree of actionability.
This is an interaction sweet spot because you can get good information in front of the
person and get attention but then also action it whether on the watch or through an
immediate secondary device like a phone or a computer.
image (cc): http://www.flickr.com/photos/ivva/7157969502/

15. Possible contender?
(c) Samsung
In my view Samsung’s prototype shown last week misses
the mark here because they’ve made it a not-quite
phone and it’s big and ugly. A smart watch doesn’t need
to be massive because it’s not a primary interface. It
doesn’t need all the features of the phone.
The winner in this space will be smart, beautiful and not
a phone wannabe.
image (c) Samsung

16. Interactive spaces
finally we have interactive spaces.
As spaces become more connected they will give us data,
visualisation opportunities and become more interactive.
This will raise some interesting challenges around
control and visualisation but also things like privacy.

17. Novel interfaces
ajfisher Lab at Rockwell
Similarly how do you take your content and display it at
the size of a building?
We’re seeing the first versions of this as tech demos, art
installations and single products but expect to see this
ramp up significantly over the next few years.
image left (cc) ajfisher
image right (c) lab at rockwell http://lab.rockwellgroup.com/work/plug-in-play.html

18. How to prepare
So how do we prepare for these insane levels of
fragmentation are going to occur - we already have to
cover most of these? What resolution is a building?
The main way to start is by having excellent content and
application structure.

19. Atomise your content
Breaking your content up atomically is critical. This
allows it to become portable and unbinds it from your
display methods
If this sounds familiar it’s because it is. This is content
first design. The natural outcome of designing content
first means it becomes a lot more portable.

20. Content choreography
You must consider content choreography so you can
ensure the relationships between your content atoms are
maintained.

21. Content sets
S M L XL
Finally, you’ll need to think about content sets which I won’t go into too much
tonight.
Conceptually these are like image sets but applied to all of your other content as well.
You need to consider what the content atom looks and behaves like as it changes
from through small, medium, large and extra large contexts. How do aspects of the
content get prioritised within each of these contexts?
This means our content responds much more cleanly across devices and can scale
from the micro to the massive much more easily and we can finally decouple
ourselves from specific device resolutions.

22. Contextual APIs
As we start thinking about content in these terms it
begins to look more and more like a contextual API.
A good current example of this is Gmail’s Actions
system. The way you interact with it varies across device
contexts but the purpose remains consistent and it
doesn’t take much to imagine this being extremely
useful on a smart watch for example..

23. New interactions
flickr (cc) Mary Scheirer
Now we’ve looked at new devices, what is happening
interaction wise?
We are now at a point where touch is a fairly legitimate
input method, though strangely we’re doing it quite
poorly on the web.
image: h"p://www.flickr.com/photos/maryscheirer/4906553164/

24. Touch
Touch is slowly starting to make us rethink the way we
design interactions because a lot of our standard
interaction methods are being removed such as the
hover context, but there’s still plenty of navigation
systems that rely on it.

25. Voice
flickr (cc) Brett Jordan
We’re on the cusp of voice being useful. It’s now
accurate enough to do useful things like search, create
reminders or get directions however most people get a
bit self conscious about using it in public.
How many of you have used voice for real though?
A handful - even a couple of years ago that probably
would have been zero and even if there was one they
would have been too embarrassed to put their hand up.
image (cc): http://www.flickr.com/photos/x1brett/8464723074

26. Gesture
flickr (cc) Brett Jordan
Gesture is getting better too.
The Wii made it culturally acceptable, Microsoft made the
technology work without a controller and now
LeapMotion, PrimeSense and others are making it
affordable and useful everywhere.

27. Gestural resolution
(c) Thalmic Labs
We’ve also got wacky products like Myo coming down
the line as well which will increase the range of gestures
we will be able to reliably detect.
Video: (c) Thalmic Labs: https://www.thalmic.com/en/myo/

28. Make this better
So what can we do to prepare?
The first thing is that as web people we need to get so
much better at touch in a composite setting. That means
when touch and traditional inputs overlap on the same
experience such as viewing a full “desktop” site on a
tablet that doesn’t have a pointer interaction method.

29. Do this now
Stay in line with the standards
Design for bigger target areas
Unbind UI events from element events
Everything you build needs to understand touch events
now and migrate to the newer standards as they become
viable.
But in addition from a design perspective, go for
naturally bigger targets, look at things like quick touches
triggering a hover state.
Consider having libraries of effects for example popping
a menu but then multiple interaction methods that do it.
Stop binding UI events directly to the element events that
trigger them.

30. Zero computer users
flickr (cc) Johan Larsson
We must get better at this than we are now.
Go get an iPad for a day and use it to do absolutely
EVERYTHING. This is your user in another 18 months.
See how bad the experience is and do something about
it.
image (cc): http://www.flickr.com/photos/johanl/5619897608/

31. Use your voice
Google
With voice, Google has an excellent speech recognition
API. Play with it. Look at how you can enhance things like
site search using speech recognition. This will help make
your site more accessible - particularly if someone is
physically impaired.
Don’t forget that even mobile users find typing difficult
and error prone.

32. Wave your arms
LeapMotion
Gesture is a bit harder because it’s tiring for the user.
However when we’re dealing with larger displays such as
TVs or buildings, gesture can be extremely intuitive as
an interaction method.
Get a Kinect or a leap motion and start playing with it.
Image (c): leapmotion.com

33. What experiences?
So what types of experiences can we create?
Each one of touch, voice and gesture has limitations on
their own but combined they become more useful and
natural.

34. Combo multipliers
For example you can use touch or gesture to group
things together and say “Send these to John”.
The combination of these affordances becomes more
useful especially in not-quite-traditional computing
contexts such as controlling a large display.

35. Retail
With devices like a leap motion, swishing through a
product list becomes very simple just by swiping your
hand and not too tiring. Other gestures could trigger
content enhancements such as popping additional
product details for example or throwing things into your
shopping cart.
So there’s a lot of potential to be had by overlapping new
interaction methods.

36. Computing substrate
flickr (cc) Wonderlane
At the outer edge of out time period we start to look at
computronium or computing as substrate. This creates
extremely elastic and slippery user experiences as the
notion of when you’re computing breaks down heavily.
Image: http://www.flickr.com/photos/wonderlane/3198166347

37. Too many devices
One of my pet annoyances is I have all these devices. I
use all of these every single day and that’s not counting
the iPhone I took the picture with.
Many of them do functionally similar things but some are
better at certain tasks than others.
The only thing making this marginally better is that I can
install chrome on some which gives me a degree of
consistency and portability from one device to another.
Devices per user are beginning to proliferate very
rapidly.

38. Resizable screens
Samsung Flexible OLED concept
So as a result we bump into another techno-cultural
meme which is the resizing display - the elastic piece of
glass. And there are people legitimately researching this
tech.
But it misses the point.
video: http://www.youtube.com/watch?v=f4AhTiQkWwk

39. Context drives uses
Flickr (cc): Henry Stratford
It’s the context shift that makes us want a different
experience. When you’re standing on a crowded tram, a
phone is perfectly useful. When I’m on a sofa a tablet is
better.
I can send an email from either of these without too
much trouble but if I sit down to write a chapter of a
book I really do need a bigger screen, a windowing
environment and definitely a keyboard.
But why do these have to be fundamentally different
devices?
image (cc): h"p://www.flickr.com/photos/henry_stradford/5348910688/sizes/o/

40. Mobile at the centre
Why don’t these other devices just provide additional
affordances to my primary computing device - which
let’s face it, is going to be my mobile phone?

41. Ubuntu Edge
(c) Ubuntu / Canonical
Ubuntu Edge - while it failed as a kickstarter - was
heading in this way where the phone became the centre.
We don’t NEED multiple discrete devices that mostly do
the same thing. However, our phone doesn’t need to BE
a PC, it just needs to be able to use the resources of the
PC when we want it to.
image (c) canonical

42. Chromecast
(c) Google
Subsumption of environmental resources is a way off yet
but I think Chrome is starting to show the way here.
Think about multi-device sync and chrome cast. These
are subsumption technologies.
Chrome doesn’t care about the underlying hardware, it
just moves the experience where the user wants it to go.
And this is just the start of this trend with the worst
technology to do it.

43. How to prepare
So how do we prepare for this world?
Mainly we start by considering how our applications can
change based on affordances presented to them that are
changing. We need to consider state changes and make
them work much much better.

44. Detecting context shifts
How do I know when someone has moved from a high
bandwidth connection to a low one? How do I know if
they’ve shifted from a touch context to a pointed one?
How do I know if someone has moved from a small
resolution to a large one AFTER the page render has
occurred?
We need to start thinking about responding to these
context shifts because they will be common sooner than
you think.

45. Retail
Retail has great potential here for me.
I can start an experience on my phone when I’m on the
tram. I come home and it can shift to my TV so I get the
benefit of a bigger display. At this point I can use
gesture to start navigating through products and adding
them to my cart.
Finally I use voice, or my onscreen or physical keyboard
to put in my details and make a purchase. This is what a
moving experience starts to become, a frictionless
transition from my set of interaction methods to another
as required by the user.

46. Continued change
So this was my view on the next handful of years.
We know will see a huge proliferation of devices leading
to a lot of fragmentation.
Gesture, voice and touch will become more common and
preferred as interaction methods.
Finally we’ll see capability being subsumed by primary
devices that will enable movable and elastic experiences.

47. Humanist technology
flickr (cc) Ding Yuin Shan
Of course, none of this is set in stone, but one thing is
certain: the web will be the main vehicle for this to occur.
The web is the only scaleable set of technologies that
has connected a billion people and is capable of
connecting 6 billion more.
However, our notions of what is a web browser are
breaking down and even native apps still consume web
technologies. The web has never been a monoculture
and the proliferation of devices, interaction forms and
technologies will create an even more diverse ecosystem.
image (cc): http://www.flickr.com/photos/90461913@N00/7901102136/

48. Hard work is coming
flickr (cc) spinster cardigan
I can’t say the next handful of years will be easy as a
designer or a developer. The shift to responsive design
as a result of mobile has shown very clearly that the
appetite for change is waning - particularly in the
corporate world.
However these technologies are coming and they will
reshape our behaviour. In 8 years we went from looking
through to looking at to take a picture. Human behaviour
is shifting rapidly, and we either embrace that or get left
behind.
image (cc): http://www.flickr.com/photos/84906483@N08/8747337118

49. Designing a
Moving Experience
Andrew Fisher @ajfisher
Be Responsive, 10 September 2013
Thankyou.
Get in touch:
Andrew Fisher (CTO JBA)
andrew@jbadigital.com
http://twitter.com/ajfisher
http://ajfisher.me/