Nippon lugano multitouch

  • 90 views
Uploaded on

Description of a multi-touch multi-user application developed for Microsoft Surface 2.0 which allows to browse exhibitions and masterpieces of the Nippon Museum of Lugano in an easy-to-use and …

Description of a multi-touch multi-user application developed for Microsoft Surface 2.0 which allows to browse exhibitions and masterpieces of the Nippon Museum of Lugano in an easy-to-use and intuitive way.

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
90
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
1
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. POLITECNICO DI MILANO Dipartimento di Elettronica e Informazione HUMAN COMPUTER INTERACTION NipponLugano Multitouch Lucio Bordonaro, Silvia Zicca, Andres Fernando Arroyave Yepes Academic Year 2011/2012
  • 2. Abstract Multitouch technologies are evolving quite fast, not only for single-user applications, but also for cooper- ative applications which allows users to interact with each others creating an engaging environment and achieving tasks together. It is quite easy to learn how to use this kind of new technology even for people that are not used at all and it allow to explore a new way of interaction that is more natural with respect to the one of the personal computer, for example. And this is why we developed a multitouch cooperative multi-user application which allows to navigate the contents of a museum presented in an innovative way and that can be easily used also by children and old people who want to know information about the exhibitions presented in the museum. 2
  • 3. Contents 1 Introduction 4 2 Requirements 5 3 Design 6 3.1 Gestures design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.2 Screens design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.2.1 Main screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.2.2 Exhibition screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3.2.3 Themes screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.2.4 Gallery screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.2.5 Image details screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.3 States diagram . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.4 Scenarios . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.4.1 A relevant non-cooperative scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.4.2 A relevant cooperative scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 4 Prototype Implementation 23 4.1 Paper prototype . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 4.2 Hardware and software architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2.1 Windows 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2.2 Vision system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2.3 Presentation layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2.4 Core layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.2.5 Windows integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.2.6 Surface shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.3 Technical requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.4 Actual state of the implementation and open issues . . . . . . . . . . . . . . . . . . . . . . . . 29 4.5 Technical problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 5 Conclusions 30 3
  • 4. 1 Introduction NipponLugano Multitouch is an application entirely dedicated to the Nippon Museum of Lugano, in Switzerland. In this museum there are four exhibitions related to Japanese photographers. That mu- seum proposes some informative contents in its website in an hypertextual and multimedial way to allow guests to deepen the cultural context to which it is dedicated to, either before or after the visit. The aim of the application is to repropose these contents in an innovative way towards a more natural interaction exploiting the new technology offered by Microsoft Surface 2.0, a multitouch surface of the last generation which allows, among other things, to perform cooperative tasks between users and to recognize the direction of the touches. The idea is to put the Surface with our application, inside the museum, near the entrance, so that po- tential guests of every age are tempted to buy a ticket after viewing a preview of the contents presented in the museum. This is because it is very easy to understand how to interact with the Surface so even a person who does not know how to use a computer, or a child can use it. This application can be used either in a cooperative way (an user can pass some contents to another user to show it to him/her and discuss together) or in a non cooperative way (in case, for example, the Surface is used by people who do not know each other or do not need to exchange information). Given the dimensions of the surface, the idea is to have users only along the long edges of the Surface. 4
  • 5. 2 Requirements This application will have only one kind of users: the potential guests of the museum. We assume that they received the practical information (address, opening hours and so on) through the website and they are just arrived to the museum. We can suppose that this big group is heterogeneous and composed by different kind of users (children in a school trip, old people, professionals photographers and also common people with no particular characteristics), but they have the same requirements and needs. In particular, considering the needs of every single independent user, he/she should receive the answer to some questions like: • what are the exhibitions of this museum? • what is every exhibition about? • what are the themes faced by each exhibition? • what are the photos related to a specified theme of the exhibition? • what are the details of a selected photo? For each one of the answers to the questions above, there will be an answer in one screen of our applica- tion which is structured to follow the questions that the user may have in order. Besides these needs, there is also the possibility to use the Surface cooperatively. So every content can be rotated and passed to another user that is placed on the other side of the Surface to discuss it together and showing it to other users. NOTE: In this documents we use the words “screens” and “windows” as a synonymous. 5
  • 6. 3 Design 3.1 Gestures design Microsoft Surface 2.0 supports a huge variety of possible gestures, so we had to select which ones to use in our application. Here we report the gestures used to interact for the Surface: • Move the windows: One or more fingers on a window to move or flick it. Figure 1: Move the window • Rotate the windows: there are three ways to rotate a window: – Single finger rotate: One finger touches a window and drags it around in a circle so that it rotates. Figure 2: Window rotation (1) – Two-finger rotate: Two or more fingers on a window are dragged in opposite directions along an arc. 6
  • 7. Figure 3: Window rotation (2) – Pin turn: One finger remains stationary acting as a pivot point while other fingers move around it. Figure 4: Window rotation (3) • Screen simple touch: a simple tap on the screen with a finger. Figure 5: Screen simple touch • Zoom in and out: Two or more fingers on an item are dragged apart (used to resize the gallery window). 7
  • 8. Figure 6: Zoom in and out 8
  • 9. 3.2 Screens design 3.2.1 Main screen The main screen consists into a red circle on a white background which represents the Japanese flag (because the museum is related to Japanese photographers). The red circle in the middle is interactive and it is split into four slices that represents the different exhibitions (identified ). The users can rotate it around its center and select the desired exhibition tapping on the slices. The use of a circle allows the users to read it easily independently from his own position with respect to the surface. Figure 7: Main screen 9
  • 10. 3.2.2 Exhibition screen Once the user selected an exhibition, it will appear a screen with all the details related to that exhibi- tion (title, author, description and some random photos). The window can be rotated so that it can be “passed” to other users that want to read it, it can be moved wherever on the surface, it can be zoomed in and out, it can be closed or can allow to open the themes screen to allow to read more about the main themes of the selected exhibition. Figure 8: Exhibition screen 10
  • 11. 3.2.3 Themes screen The themes screen will show information about the themes of the selected exhibition (title, description and one representative image). This window has a circular menu. Rotating it, it is possible to view in- formation about the different themes. Tapping on the representative image, the gallery of the theme will open. This screen can be moved and it is possible to show or hide the description of each theme. Figure 9: Themes screen 11
  • 12. 3.2.4 Gallery screen Every theme has a gallery of images. This gallery window can be enlarged or reduced. Also the images contained into the window can be zoomed in and out and is possible to scroll horizontally and vertically to view images not directly showed when the screen is too small. Figure 10: Gallery screen 12
  • 13. 3.2.5 Image details screen Every time you tap an image from a gallery or the exhibition library bar, it will open in an independent screen with related details (title, author and description). It is possible for the user to pass the image to another user without passing him/her the whole gallery of that theme. It is also possible to move, rotate and zoom the image. Text can be showed or hidden. Figure 11: Image details screen 13
  • 14. 3.3 States diagram The following diagram shows all the possible interactions that can be done with each window and how to pass from one to another. Consider also that every window can be closed by the user in every moment without influencing the behavior of the other. Also they can be opened more than ones simultaneously by the two side of the surface. Figure 12: State diagram 14
  • 15. 3.4 Scenarios 3.4.1 A relevant non-cooperative scenario Paolo wants to visit the Nippon Museum following the advice of a friend who has already gone there and who has talked him vaguely what it is about. When he arrives to the museum, he has never visited the website, so he knows only the practical information received by his friend (opening hours, address and ticket price), so he decides to use the Surface to gain more detailed information about the contents of the museum before buying the ticket. From the main screen of the application, Paolo decides to retrieve information about the Araki’s exhibi- tion. Figure 13: Araki exhibition 15
  • 16. After he read about that exhibition and slided some of the images it contains, he decides see what are the themes faced in that exhibition. Figure 14: Araki themes 16
  • 17. Chosen the theme Artistic Expression, Paolo decides to browse its gallery to have an idea about what photos he will find about that theme. Figure 15: Araki theme: Artistic Expression 17
  • 18. While zooming the gallery, Paolo is impressed by a photo, so he decides to zoom it in and to read some additional information about it. Figure 16: Araki photo details Then, Paolo decides to close all the screens and to buy a ticket of the museum to visit it. 18
  • 19. 3.4.2 A relevant cooperative scenario Marta and Sara are two sisters fond of photography who decide to visit together the Nippon Museum of Lugano. They walk towards the Surface and they place along the two long sides of the Surface and start browsing information about two different exhibitions: Araki (Sara) and Shunga (Marta). Figure 17: Multiuser exhibition 19
  • 20. While Marta is still reading, Sara opens the themes screen of Araki exhibition, she select the one she prefers opening the gallery. Figure 18: Multi-user 20
  • 21. Marta closes the Shunga screen and decides to view information about Ineffable Perfection, meanwhile Sara opens and zooms in a photo that she likes. Figure 19: Opening a new exhibition 21
  • 22. Sara passes the zoomed photo to her sister to listen to her opinion. Figure 20: Passing content to another user They are both interested into that particular photo, so they decide to go to buy the tickets to watch it. 22
  • 23. 4 Prototype Implementation 4.1 Paper prototype After deciding which are the main elements of the application, we built up a paper prototype to simulate the behavior of each component and analyzing if the user requirements were respected. As the term “prototype” says itself, the definitive version of the application presents some differences. Here follows some photos about the paper prototype. Figure 21: Exhibition selector Figure 22: Exhibition screen 23
  • 24. Figure 23: Themes screen Figure 24: Gallery screen 24
  • 25. Figure 25: Image details screen 25
  • 26. 4.2 Hardware and software architecture The following figure represents the entire Surface platform, including the hardware layer and software components that you must be aware of when you develop Surface applications. Figure 26: Software architecture 4.2.1 Windows 7 Microsoft Surface 2.0 runs on the Windows 7 64-bit operating system. Windows 7 provides all the ad- ministrative, security, and directory functionality of the Surface. Developers and administrators working with a Microsoft Surface unit have full access to Windows functionality (in Windows Mode). However, when people interact with Microsoft Surface applications, the Windows user interface is suppressed (in Surface Mode). 4.2.2 Vision system The Vision System uses PixelSense™ to process captured touch data into useful application data you can access through Surface SDK APIs. PixelSense™ enables each pixel in the Surface display to detect when a person touches it or when someone moves a finger, tagged object, or untagged object over it. It does this without the use of cameras, which is what makes newer Surface hardware much thinner than previous versions. 4.2.3 Presentation layer The Presentation layer integrates with Windows Presentation Foundation (WPF) and includes a suite of interaction controls designed for Microsoft Surface enabling you to quickly and easily build touch enabled applications. 26
  • 27. 4.2.4 Core layer The Core layer exposes Microsoft Surface specific contact data and events so you can create Microsoft Surface enabled applications with any user interface (UI) framework that is based on HWND (Handle WiNDows). 4.2.5 Windows integration The tight integration between Microsoft Surface and the Windows operating system provides system wide functionality on top of the Windows operating system. You must use this functionality to support unique aspects of the Microsoft Surface experience, such as managing user sessions, switching between the standard Windows user interface (Windows Mode) and the deployment experience (Surface mode), monitoring critical Microsoft Surface processes, and handling critical failures. 4.2.6 Surface shell Surface Shell is the component that manages applications, windows, orientation, and user sessions and provides other functionality. Every Microsoft Surface application must integrate with Surface Shell. 27
  • 28. 4.3 Technical requirements To install and develop applications using the Microsoft Surface 2.0 SDK, you must you have the following software installed on your Windows 7 PC workstation, Windows 7 touch PC, or Surface unit that you are using for development: • Windows 7 operating system (64-bit recommended) • .NET Framework 4.0 • Visual Studio 2010 or Visual C# 2010 • XNA 4.0 Redistributable or XNA Games Studio 4.0 • Microsoft Expression Blend 4.0 (recommended) Additionally, you must create your applications to run on hardware designed for Surface 2.0, such as the Samsung SUR40 for Microsoft Surface, or Windows 7 touch PCs. 28
  • 29. 4.4 Actual state of the implementation and open issues At the moment the application responds to all the user’s requirements specified in Chapter 2, but it is only a prototype that needs to be tested with real users. Also there are still some open issues to fix. First of all, the gallery of each exhibition is made to contain nine images related to the exhibition. The number of nine was chosen because it is simple to dispose images in a 3x3 matrix and put in the middle of the square the selected image, so that the user can navigate all the figures starting from the chosen one. Also it is not a big number, so it does not overload memory and slow down the program execution. We decided to show different images from the same exhibition and the simplest way to do this was to chose one image for each theme of the exhibition. The problem is that some exhibitions do not have nine themes (but they have eight) so there can be some images repeated more than once. The same problem is found when opening the gallery of a specified theme of the chosen exhibition: some themes do not have nine images, so there are duplicates. We analyzed the problem and one solution was to leave some empty spaces, but we decided that put duplicates was a better solution, because empty spaces could have confused the users. Another possibil- ity was to completely modify the structure of the gallery, but doing this we should have changed all the possible interactions and GUI related to that screen. Another issue consists in the fact that in the actual state of the implementation we have not set a max- imum number of screens that can be opened on the Surface simultaneously. Probably it is not a real problem since the Surface automatically put the last touched window over the others. Also we have not set the maximum dimensions that every screen can have. 4.5 Technical problems The main problem we had to face during the implementation part of the project, is that Microsoft Surface 2.0 was not on the market at the moment in which we developed the application, so we had to test it on normal laptop with a screen resolution which is not like the one of the real surface (that is full HD). Also we had to use a touch simulator to simulate multitouch actions and other kind of interactions that are not easily created using mouse movements. Another minor problem was that some guidelines find on the Surface official website for developer, are too strict regards some aspects (i.e. the font size). 29
  • 30. 5 Conclusions The project NipponLugano Multitouch is a useful application related to Nippon Museum of Lugano, but it can be easily modified and adapted to every kind of museum of exhibition losing only the metaphor of the Japanese flag in the main screen. The main thing we learned is to work in contact with new technologies and in particular with multitouch world which offers a huge number of interactions and possibilities that you cannot find in classical com- puters. Also the Surface allows the simultaneous interaction of more than one users on the same device and this cannot be provided by normal small devices with touch screen (i.e. smartphones, tablets...). Working on this kind of technology, means to think in a totally different way about the human-computer interaction and focus on the more natural interactions and gestures to allow the users to focus themselves on the contents and not on what they have to do to find and manipulate it. 30