Designing a Moving Experience


Published on

It’s 2013. The shift to mobile is well and truly upon us, we’re at the transition point where Mobile Internet overtakes Desktop Internet usage and there is no going back. If you’re not designing responsively now then you better get cracking because what comes next is a big change to the way we design and build web experiences for humans.

The shift to mobile started making us think about devices, however almost all of our discussion is around what devices we support and where do we set our breakpoints. Could this focus on device capability be masking something bigger happening in the way humans are starting to behave with the web?

What if the fragmentation we’re seeing on Android is merely a glimpse into the device fragmentation of the future? What happens when a users’ experience can range from the interface of a watch to that of a building? How do we communicate with someone when they are walking down the street trying to locate something compared to kicking back on the sofa? How do we even know when they are doing one and not the other?

Contextually Responsive Design is quickly going to become a necessity - but this isn’t personalisation 2.0 or Content First under a different name. To truly design engaging experiences we need to consider how context shapes our behaviour. As Web Designers and Developers we’ve traditionally worked with people “sitting down to compute”. What happens when someone no longer wants to sit down at a computer to do something; ever?

This talk will start from the point of late 2013 and look forward six years. How do we expect people to behave? How will we design systems to cope, and what are the contexts in which people will use the web during its next age?

Given at Melbourne Be Responsive on 10 September 2013.

Published in: Design, Technology
  • Be the first to comment

Designing a Moving Experience

  1. 1. Designing a Moving Experience Andrew Fisher @ajfisher Be Responsive, 10 September 2013 Hi! My name is Andrew Fisher and I’m an interaction researcher. Tonight I’m going to talk to you about how to design a moving experience for the next 25 minutes.
  2. 2. 8 years difference I imagine everyone has seen this picture by now. In this 8 years between popes it shows a striking technological change. However what’s interesting for me is that this 8 year period depicts a fundamental change in behaviour too.
  3. 3. 2005 Reuters / Jerry Lampen This pic shows it better. Look down here at what people are holding... Cameras you need to LOOK through to see what you’re shooting. image:
  4. 4. 2013 AP Here, there are more cameras because there are more phones. And the one person with an iPad. But we’ve moved from look through to look at because now EVERYTHING has a screen. In the span of 8 years, the behaviour of hundreds of millions of people has changed irreversibly after being culturally ingrained for about a hundred years. These pics show how behaviour is changing and is changed by new interaction types afforded by our devices. image:
  5. 5. Time at human scale Flickr (cc): Pink Sherbet Photography So apparently the average span of a pope is about six and a half years. As a timespan it’s nice to work with as it doesn’t feel as long as a decade. It’s the division of time we use when we grow up that covers pre school, junior school and senior school. It’s a very human time scale and it’s also long enough to see changes happen. image (cc):
  6. 6. Moore’s Law 7 years also represents about a 10x increase in electronic capability according to Moore’s law. So this time period is a very interesting one to observe for many reasons.
  7. 7. Not many years ago... flickr (cc) Zitona 6-7 years ago we didn’t have iPhones or Android phones. We barely had voice recognition at all, let alone on our phones. The nintendo Wii was just showing us a future where waving your arms around at a machine wasn’t a sign of madness but something quite normal people would be happy to do. But I feel we’ve gone off track. Responsive design has focussed too much on our devices and on technical solutions and too little on the behaviour that goes alongside. image: (cc)
  8. 8. Design for humans flickr (cc) jonrawlinson So let’s look forward and see what the next 6 years will bring us and how we will design moving experiences in the future. Tonight I want you to think about how responsive design goes beyond just our technology. About how we design for humans and their behaviour not the devices themselves so that experiences are more contextually aware. To do that we are going to cover: image (cc):
  9. 9. Agenda 1. New device types 2. New interaction methods 3. Computing as substrate How new device types will influence our design. What new methods of interaction should we consider And how computing will become embedded into our environment.
  10. 10. New devices flickr (cc) x-ray delta one Lets start with new devices coming up over the next handful of years. What will they do? How will they change our behaviour? image (cc):
  11. 11. Glass-like flickr (cc) guiseppe.costantino Google is betting the future on Glass and there’s a lot of noise about this right now because it taps into an Augmented Reality meme going back at least as far as terminator. image (cc):
  12. 12. Niche applications flickr (cc) loiclemeur There will be some very good, although highly niche use cases for glass like devices. Though I can’t see them being used all the time as they are intrusive and likely to be psychologically draining. image (cc): 8699901706/
  13. 13. Retail applications flickr (cc) Zemlinki! Mach 4 X-Trainer $189.99 ★★★★☆ Other prices: $149.99 (Amazon) $134.56 (Shoes) I can see good use cases in retail, where you pick up a product and shows us competitive pricing, features and customer ratings. There’s some domain specific benefits here but they mostly already work on mobile however they give you the benefit of efficiency because you hands become more free. In this time span there’s likely to be a lot of hype but a lot of disillusionment as well. image (cc):
  14. 14. Microscreens flickr (cc) Ivva! Next up are microscreens and in particular smart watches. These are much closer to reality and there are plenty of companies playing here right now with products or prototypes. Smart watches lay in an interesting area of ambient technology at the intersection of relatively high information density and a high degree of actionability. This is an interaction sweet spot because you can get good information in front of the person and get attention but then also action it whether on the watch or through an immediate secondary device like a phone or a computer. image (cc):
  15. 15. Possible contender? (c) Samsung In my view Samsung’s prototype shown last week misses the mark here because they’ve made it a not-quite phone and it’s big and ugly. A smart watch doesn’t need to be massive because it’s not a primary interface. It doesn’t need all the features of the phone. The winner in this space will be smart, beautiful and not a phone wannabe. image (c) Samsung
  16. 16. Interactive spaces finally we have interactive spaces. As spaces become more connected they will give us data, visualisation opportunities and become more interactive. This will raise some interesting challenges around control and visualisation but also things like privacy.
  17. 17. Novel interfaces ajfisher Lab at Rockwell Similarly how do you take your content and display it at the size of a building? We’re seeing the first versions of this as tech demos, art installations and single products but expect to see this ramp up significantly over the next few years. image left (cc) ajfisher image right (c) lab at rockwell
  18. 18. How to prepare So how do we prepare for these insane levels of fragmentation are going to occur - we already have to cover most of these? What resolution is a building? The main way to start is by having excellent content and application structure.
  19. 19. Atomise your content Breaking your content up atomically is critical. This allows it to become portable and unbinds it from your display methods If this sounds familiar it’s because it is. This is content first design. The natural outcome of designing content first means it becomes a lot more portable.
  20. 20. Content choreography You must consider content choreography so you can ensure the relationships between your content atoms are maintained.
  21. 21. Content sets S M L XL Finally, you’ll need to think about content sets which I won’t go into too much tonight. Conceptually these are like image sets but applied to all of your other content as well. You need to consider what the content atom looks and behaves like as it changes from through small, medium, large and extra large contexts. How do aspects of the content get prioritised within each of these contexts? This means our content responds much more cleanly across devices and can scale from the micro to the massive much more easily and we can finally decouple ourselves from specific device resolutions.
  22. 22. Contextual APIs As we start thinking about content in these terms it begins to look more and more like a contextual API. A good current example of this is Gmail’s Actions system. The way you interact with it varies across device contexts but the purpose remains consistent and it doesn’t take much to imagine this being extremely useful on a smart watch for example..
  23. 23. New interactions flickr (cc) Mary Scheirer Now we’ve looked at new devices, what is happening interaction wise? We are now at a point where touch is a fairly legitimate input method, though strangely we’re doing it quite poorly on the web. image: h"p://
  24. 24. Touch Touch is slowly starting to make us rethink the way we design interactions because a lot of our standard interaction methods are being removed such as the hover context, but there’s still plenty of navigation systems that rely on it.
  25. 25. Voice flickr (cc) Brett Jordan We’re on the cusp of voice being useful. It’s now accurate enough to do useful things like search, create reminders or get directions however most people get a bit self conscious about using it in public. How many of you have used voice for real though? A handful - even a couple of years ago that probably would have been zero and even if there was one they would have been too embarrassed to put their hand up. image (cc):
  26. 26. Gesture flickr (cc) Brett Jordan Gesture is getting better too. The Wii made it culturally acceptable, Microsoft made the technology work without a controller and now LeapMotion, PrimeSense and others are making it affordable and useful everywhere.
  27. 27. Gestural resolution (c) Thalmic Labs We’ve also got wacky products like Myo coming down the line as well which will increase the range of gestures we will be able to reliably detect. Video: (c) Thalmic Labs:
  28. 28. Make this better So what can we do to prepare? The first thing is that as web people we need to get so much better at touch in a composite setting. That means when touch and traditional inputs overlap on the same experience such as viewing a full “desktop” site on a tablet that doesn’t have a pointer interaction method.
  29. 29. Do this now Stay in line with the standards Design for bigger target areas Unbind UI events from element events Everything you build needs to understand touch events now and migrate to the newer standards as they become viable. But in addition from a design perspective, go for naturally bigger targets, look at things like quick touches triggering a hover state. Consider having libraries of effects for example popping a menu but then multiple interaction methods that do it. Stop binding UI events directly to the element events that trigger them.
  30. 30. Zero computer users flickr (cc) Johan Larsson We must get better at this than we are now. Go get an iPad for a day and use it to do absolutely EVERYTHING. This is your user in another 18 months. See how bad the experience is and do something about it. image (cc):
  31. 31. Use your voice Google With voice, Google has an excellent speech recognition API. Play with it. Look at how you can enhance things like site search using speech recognition. This will help make your site more accessible - particularly if someone is physically impaired. Don’t forget that even mobile users find typing difficult and error prone.
  32. 32. Wave your arms LeapMotion Gesture is a bit harder because it’s tiring for the user. However when we’re dealing with larger displays such as TVs or buildings, gesture can be extremely intuitive as an interaction method. Get a Kinect or a leap motion and start playing with it. Image (c):
  33. 33. What experiences? So what types of experiences can we create? Each one of touch, voice and gesture has limitations on their own but combined they become more useful and natural.
  34. 34. Combo multipliers For example you can use touch or gesture to group things together and say “Send these to John”. The combination of these affordances becomes more useful especially in not-quite-traditional computing contexts such as controlling a large display.
  35. 35. Retail With devices like a leap motion, swishing through a product list becomes very simple just by swiping your hand and not too tiring. Other gestures could trigger content enhancements such as popping additional product details for example or throwing things into your shopping cart. So there’s a lot of potential to be had by overlapping new interaction methods.
  36. 36. Computing substrate flickr (cc) Wonderlane At the outer edge of out time period we start to look at computronium or computing as substrate. This creates extremely elastic and slippery user experiences as the notion of when you’re computing breaks down heavily. Image:
  37. 37. Too many devices One of my pet annoyances is I have all these devices. I use all of these every single day and that’s not counting the iPhone I took the picture with. Many of them do functionally similar things but some are better at certain tasks than others. The only thing making this marginally better is that I can install chrome on some which gives me a degree of consistency and portability from one device to another. Devices per user are beginning to proliferate very rapidly.
  38. 38. Resizable screens Samsung Flexible OLED concept So as a result we bump into another techno-cultural meme which is the resizing display - the elastic piece of glass. And there are people legitimately researching this tech. But it misses the point. video:
  39. 39. Context drives uses Flickr (cc): Henry Stratford It’s the context shift that makes us want a different experience. When you’re standing on a crowded tram, a phone is perfectly useful. When I’m on a sofa a tablet is better. I can send an email from either of these without too much trouble but if I sit down to write a chapter of a book I really do need a bigger screen, a windowing environment and definitely a keyboard. But why do these have to be fundamentally different devices? image (cc): h"p://
  40. 40. Mobile at the centre Why don’t these other devices just provide additional affordances to my primary computing device - which let’s face it, is going to be my mobile phone?
  41. 41. Ubuntu Edge (c) Ubuntu / Canonical Ubuntu Edge - while it failed as a kickstarter - was heading in this way where the phone became the centre. We don’t NEED multiple discrete devices that mostly do the same thing. However, our phone doesn’t need to BE a PC, it just needs to be able to use the resources of the PC when we want it to. image (c) canonical
  42. 42. Chromecast (c) Google Subsumption of environmental resources is a way off yet but I think Chrome is starting to show the way here. Think about multi-device sync and chrome cast. These are subsumption technologies. Chrome doesn’t care about the underlying hardware, it just moves the experience where the user wants it to go. And this is just the start of this trend with the worst technology to do it.
  43. 43. How to prepare So how do we prepare for this world? Mainly we start by considering how our applications can change based on affordances presented to them that are changing. We need to consider state changes and make them work much much better.
  44. 44. Detecting context shifts How do I know when someone has moved from a high bandwidth connection to a low one? How do I know if they’ve shifted from a touch context to a pointed one? How do I know if someone has moved from a small resolution to a large one AFTER the page render has occurred? We need to start thinking about responding to these context shifts because they will be common sooner than you think.
  45. 45. Retail Retail has great potential here for me. I can start an experience on my phone when I’m on the tram. I come home and it can shift to my TV so I get the benefit of a bigger display. At this point I can use gesture to start navigating through products and adding them to my cart. Finally I use voice, or my onscreen or physical keyboard to put in my details and make a purchase. This is what a moving experience starts to become, a frictionless transition from my set of interaction methods to another as required by the user.
  46. 46. Continued change So this was my view on the next handful of years. We know will see a huge proliferation of devices leading to a lot of fragmentation. Gesture, voice and touch will become more common and preferred as interaction methods. Finally we’ll see capability being subsumed by primary devices that will enable movable and elastic experiences.
  47. 47. Humanist technology flickr (cc) Ding Yuin Shan Of course, none of this is set in stone, but one thing is certain: the web will be the main vehicle for this to occur. The web is the only scaleable set of technologies that has connected a billion people and is capable of connecting 6 billion more. However, our notions of what is a web browser are breaking down and even native apps still consume web technologies. The web has never been a monoculture and the proliferation of devices, interaction forms and technologies will create an even more diverse ecosystem. image (cc):
  48. 48. Hard work is coming flickr (cc) spinster cardigan I can’t say the next handful of years will be easy as a designer or a developer. The shift to responsive design as a result of mobile has shown very clearly that the appetite for change is waning - particularly in the corporate world. However these technologies are coming and they will reshape our behaviour. In 8 years we went from looking through to looking at to take a picture. Human behaviour is shifting rapidly, and we either embrace that or get left behind. image (cc):
  49. 49. Designing a Moving Experience Andrew Fisher @ajfisher Be Responsive, 10 September 2013 Thankyou. Get in touch: Andrew Fisher (CTO JBA)