Natural Interaction : Designing Mobile Gestural Controls (2009)
•Download as PPT, PDF•
1 like•834 views
This is a talk and the transcript given at media events organized for Nokia Design in 2009. This talk is based on a Gestures project I led while working for Service & UI Design (SUID) team in Nokia Design London. Special thanks to the project team including: Akseli Anttila, Greg Steeves, Tim Brooke, Henry Holland, Greg Edwards, Anab Jain, Juliana Ferreira, along with many other helpers in SUID (especially Tom Jenkins who models for the videos and Joe Macleod for supporting the project).
Recommended
More Related Content
Viewers also liked
Viewers also liked (12)
Recently uploaded
Recently uploaded (20)
Natural Interaction : Designing Mobile Gestural Controls (2009)
- 6. 6
Editor's Notes
- // This is a talk and the transcript given at media events organized for Nokia Design in 2009. // This talk is based on a Gestures project I led while working for Service & UI Design team in Nokia Design London. // Contact: jung@younghee.com / jabbaby@gmail.com // http://younghee.com My name is Younghee and it’s my 9th year in Nokia, specializing in explorative design research. I am originally from South Korea, and moved to London about a year ago. How many of you have traveled to Europe or America? Then you probably could sympathize with me how confusing it can be to greet someone you meet for the first time. Sometimes handshake / hug / bowing. Sometimes kissing lightly on the cheeks - There are so many different ways – especially in London, possibly because it is such an international city. My colleagues who are born in London say they are still confused too. What I am going to share today is about natural interaction. Do you remember your first encounter with a computer? My colleague’s first memory of seeing the computer was at a shop called ‘Boots’. He typed in ‘Hello’ to the computer and expected it to say ‘Hello’ back. But it didn’t, and he was quite disappointed. We probably don’t talk about ‘natural’ way to interact with people, but recently we started to talk about interacting with technology in more ‘natural ways’. Technology is now mature enough to investigate how we interact with products that we use in our everyday lives – like through gestures.
- Talking about natural – certain things come really easy to us, being human. Like facial expressions. We almost have to make conscious effort to be able to control our facial expressions not too revealing. But doing what we so ‘naturally’ do through technology can be quite difficult. This is a robotic cat, able to simulate emotional expressions. Our communication practice is quite complicated. Verbal communication plays an important part, but non-verbal side of communication, like facial expressions has substantial influence. That’s one of the reasons why people often say “you cannot replace face to face communication”, like “I don’t want to talk about this on the phone”. Companies invest a lot of money in making realistic video conferencing tools– to enable next best communication to face to face meetings.
- Certain body languages, or gestures travel really well – putting a finger in front of lips is a universal gesture. This is a picture from Taiwan taken by an American tourist. Photo: Uploaded at http://www.flickr.com/photos/libraryman/16779223/ Taipei, Taiwan.
- On the other hand, some gestures are quite culturally specific. These are two business men exchanging cards. They bow, holding cards with two hands and try to lower their hands below the other person’s – it’s a sign of respect, but it may be puzzling for those who do not understand this ritual. Photo: Uploaded at http://www.flickr.com/photos/c0t0s0d0/2011797111/ Tokyo, Japan.
- Some gestures can be important to indicate your membership, or belongingness. I was in Rio, and people around me were all doing this so I picked it up quite quickly. In this photo, I was posing with our taxi driver who drove us in and out of the favela we were working in. Later I was told that it was a sign to indicate that you belong to the local community (a specific gang). Photo: Nokia, Rio de Janeiro, Brazil. 2007.
- Some gestures travel through time. This is the V sign, which became known because Churchil during the 2nd world war meaning victory. And in the picture, this is Nixon’s V in 1960’s after he won the election. Photo from http://en.wikipedia.org/wiki/V_sign
- And not only time but also geography – these are photos recently taken in Laos and in Taipei – both kids and adults alike. The V sign in Asia lost its meaning as victory, but used more widely as a ritual for children to pose for photos.
- We saw how gestures develop in society, but sometimes new technology or popular products could create new gestures. How would you indicate ‘call me’ or ‘I will call you’? Dial phone would have been two hands signal… With mobile phone nowadays, it is something like (this) – simpler.
- Our phones have been changing a lot in the past… We already have a few products in the market which incorporates natural gesture interaction. For instance, Nokia 8800 understandings turning it facing down. In our previous studies, especially in Asia – Women often mentioned embarrassment of the phone ringing where it should not. Turning the phone over to silence it is elegant but also suitable for the situation.
- Here’s another example of gesture. On the left, it was me learning the local greeting ritual with hands, which ends with snapping fingers together in Ghana. On the right, it is a picture from user test session when people are acting out how to give something through phones, which reminded us of the handshake in Ghana. The purpose of our research project was to find natural gestures we use everyday that could be translated into how we use our devices. Technology is maturing to the point where it can understand our natural gestures. The tricky part is understanding which ones. Photo left: Greeting ritual using hands. Younghee Jung, Accra, Ghana. 2007. Photo right: A situated acting to transfer a data from one phone to another without using buttons or screen interface. Younghee Jung, Shanghai, China. 2009.
- Sample task given to research participants to come up with how they would control the phone for the given situation without using buttons or on-screen interface.
- Sample task given to research participants to come up with how they would control the phone for the given situation without using buttons or on-screen interface.
- We set up the research in four different cities – Shanghai, Mumbai, Chicago, London. We selected 11 things you can do with mobile phones – some very common like checking time, some becoming more common, like sending a photograph.
- Then we gave them dummy blocks to act out how they would use gestures to control the phone. For instance, one of the tasks was ‘silence the phone’. We asked participants to imagine the situation that the phone is ringing loudly in the office or in a quiet place. How would you silence the phone without pressing buttons? Photo: User research session in exploring gestural control of mobile phones. Younghee Jung, Shanghai, China. 2009.
- We sketched out ideas from the participants. Here are some of them – on silencing the phone. Ideas we collected vary a lot from being very pragmatic to ‘magical’ or ‘highly futuristic’. On the pragmatic side, to silence the phone, some people mentioned ‘cover the speaker with the hand’. On the ‘magical’ side – some mentioned staring at the phone to mute. Or say ‘oops or shhhhhh’. Photo: Sketches done by research participants on how they would like to control their phones. Nokia, 2009.
- We also gathered some ideas that mimic the existing cultural etiquette in the gestures. In exchanging business cards, people in China and Mumbai wanted to hold the phone with two hands in similar manner. Similar to the picture we saw earlier. Obviously that did not come out in Chicago and London. Photo: User research session in exploring gestural control of mobile phones. Younghee Jung, Shanghai, China. 2009.
- So that was the first phase of our exploration. We allowed people to freely think about how they would instinctively like to control their phones given the situations that we presented. It was mostly based on spontaneous reactions or ideas developed through acting out with their partners in research session. This first phase generated lots of ideas that people considered ‘natural’ or ‘instinctive’. Some quite obviously common, some not very intuitively clear.
- So the next phase of the research was giving a completely different scope in allowing people to think about the gestural control. We ran it in Japan as well as in London so that there is a minimal check on the culturally specific reactions. For a selected number of situations using mobile phones, we chose 4 different alternative gestures. For instance this is a task for sending a data from one phone to another. We showed 4 very short video clips of these examples to research participants.
- So the next phase of the research was giving a completely different scope in allowing people to think about the gestural control. We ran it in Japan as well as in London so that there is a minimal check on the culturally specific reactions. For a selected number of situations using mobile phones, we chose 4 different alternative gestures. For instance this is a task for sending a data from one phone to another. We showed 4 very short video clips of these examples to research participants.
- This is an example to execute a favorite function. We did include some flamboyant idea like ‘kissing the phone’ – which was an idea presented by a research participant in the first phase.
- In order to understand factors that make gestures work well or not, we had to be more systematic than the first stage. In total, we selected 60 different gesture ideas, some from the first stage, but some developed by the design team. The purpose of this stage is not to pick the best gesture ideas. Rather, we wanted to understand everyday situations and various cultural contexts from participants, which was why we had participants from 14 different countries. For instance, one of the 15 tasks we chose was sending something to another phone, such as throwing action. These examples were shown as video clip and we asked participants what they thought about them. These were not chosen because they were the right ones, but because they stimulate participants’ thinking. I am going to show some examples from the study.
- One of the very fundamental issues that most participants were concerned about was whether they can do it. Sometimes, it was about whether their body will move comfortably enough for the gesture, sometimes it was about whether they can do it accurately. Video 5.4. Shaking the phone to open an application : “I don’t want my arm to be tired using the phone.” “I won’t be able to do it in crowded places – I may hit other people doing this.” Video 3.3. Brushing hand left/right to browse images: “Requires using two hands. I want my mobile phone operable with one hand only” “I won’t use it for something I need to find quickly and accurately, but it would be nice if I were at home browsing photos, relaxed.”
- Working for a mobile phone company, we sometimes take pride in testing our products by throwing them / dropping them as they are quite robust. But our participants were quite reluctant to do anything that would risk their beloved mobile phone to be damaged. Video 11.2. Tapping the device on a surface to switch the profile: “What if I break the phone..” “Will I need a table or a flat surface to do this?” Video 14.1. Handshake with phones on palms to give something over the phone: “I don’t want to touch other people’s phone with mine. It will make screen scratched.” “I don’t want to have to touch the other person to give something through the phone. It may be someone I don’t like!”
- One of the interesting findings to me was how conscious people were about how they would be seen by others ‘performing’ the gesture. They were very aware of the various environments that they use mobile phones in. Video 6.1. Waving the phone in the air in a circle to find out where you are: “Can I draw a smaller circle, downwards? It may be difficult to do in crowded places, like bus stop.” “I don’t want to attract attention from others when I am lost. It will be like advertising to thieves ‘hey, steal my phone’!” Not wanting to show the phone in public was quite strong among participants living in London. Video 5.3. Kissing the phone to call a loved one: A lady from Malaysia who was a nurse: “I think it would be cute but I am a nurse. Kissing the phone would not be hygienic.” A lady from Turkey: “I would only kiss human beings. I don’t want to treat non-living thing as human.” Especially those who come from middle east were strongly opposed to any gesture that implies the phone to be ‘alive’ / like humans.
- We invited again various people to act out the gestures – but this time we brought them to real contexts and experience real situations, like church, busy streets, public transport, meeting rooms. This way, we could see how the gestures could really work. We used this as a filter for the ‘right gestures’. Photo: Situated validation of the gestures in the streets and other public places in London. Nokia, 2009.
- With this new knowledge, we went back to the studio and created a prototype. One part of the prototype was a tool to tweak and change the parameters of the gesture. The other part was a physical device to try out the gesture. Trying out the gesture with prototypes is crucial for developing design details, like how the visual, audio, or haptic feedback is given for the gesture. Simply no designer’s imagination can be that great to design the gesture without ‘feeling’ the gesture in the hands. Along with the design detail development, we will run more user studies with the prototype, because people’s reaction can be very different once they have something working in their hands. Photos: Prototype on the gestural controls and user research session for improving the design. Nokia, 2009
- Natural but a learnable language Gestures should work as a language together – like any sign languages that we have devised over the years. Gestural controls present an intriguing potential of the technology objects that are being so closely intertwined in our every lives. Personally, I don’t believe the technology should always be seamlessly integrated that it disappears completely in the background, but there are a lot of areas that it can be to benefit how we live with technology. .
- Earlier we started with a few examples of gestures. Some are global (be quiet)– common, some are culturally specific (politeness). It is about getting the richness of meaning, not to find the generic global gestures. Designing gesture interaction is like a little new language for everyone, and our role is to make sure that the new language is developed in such a way that can be easily adopted by most people on the planet. This ongoing study was agnostic to technological feasibility, but will guide its development. // This is a talk and the transcript given at media events organized for Nokia Design in 2009. // This talk is based on a Gestures project I led while working for Service & UI Design team in Nokia Design London. // Contact: jung@younghee.com / jabbaby@gmail.com // http://younghee.com