The times are a-changing in the interface


Published on

Presentation held at the conference "Multimodality and Learning: New Perspectives on Knowledge, Representation and Communication", London, 19-20 June 2008. More information:

Published in: Design, Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

The times are a-changing in the interface

  1. 1. The times are a-changing in the interface Andrew Morrison InterMedia, University of Oslo Oslo School of Architecture & Design and Jon Olav Eikenes Oslo School of Architecture & Design
  2. 2. Introduction
  3. 3. Social semiotics as frame (Kress & van Leeuwen) Socio-cultural perspective on design RECORD project: text-production-use study Discourse in action (Norris & Jones 2005) ‘ … multimodal refers to communicative artifacts and processes which combine various sign systems (modes) and whose production and reception calls upon the communicators to semantically interrelate all sign repertoires present.’ (Stöckl 2004: 9)
  4. 4. Interface
  5. 5. ‘movement in the interface’ (Skjulstad & Morrison 2005) multimodal, multiply, mediated activity navigational features are kinetic ‘transtextuality’, discourse across links (Kolb 1996)
  6. 6. Navimation
  7. 7. Navimation: kinesis in navigation is intertwined with animation in the interface
  8. 8. Navimation & multiliteracies • situated and multimediational multiliteracies (Snyder 1998, Ulmer 1998, Cope & Kalantzis 2000, Hawisher & Selfe 2000, Kress 2003, Buckingham 2003) • multimodal literacies in schools (Jewitt & Kress 2003, Jewitt 2005) • metamedia literacy (Lemke 1998) • part of the participatory flux of popular culture (Anderson 2003, Jenkins 2006) • social semiotic frame, desinging affordances, social networking
  9. 9. Concepts
  10. 10. Selection of sites in a social semiotic frame: • web usage, web design practice, web analysis • trawling web for instances of navimation • identification of c. 100 sites with generic navimational features • categorisation of core kinetic features (informed by concepts from interface design, hypertext, information visualisation, film, animation, motion graphics, multimodal discourse, electronic art, etc) • selection of sites with clear yet challenging features for textual analysis
  11. 11. Core concepts: • Multimodal interaction • Situated motion • Transformative motion • Non-sequential navigation • Screened spatialisation
  12. 12. Analysis
  13. 13. Multimodal interaction
  14. 14. Multimodal interaction • physical context physical mobile device in urban setting vs. stationary home PC device touchscreen vs. mouse • degree of interaction sporadic vs. continuous • form of interaction body gestures vs. direct mapping • interface (HCI): ‘any part of the computer system that the user comes in contact with’ (Moran 1981) • interaction modes (info.vis.): continuous, stepped, passive, and composite (Spence 2007) • embodied interaction (Dourish 2004) • human factors, ergonomic design, haptic design etc.
  15. 15. Situated motion
  16. 16. Situated motion • Motion that is generated and screened in real time, in the use situation • Not predefined in detail, but allowed for by the designer • Can be affected by parameters such as user interaction • Machinima (machine cinema) : production techniques where computer- generated imagery is rendered using real-time, interactive 3D engines (Burn & Parker 2001) • Software art, computer games, Virtual Reality, Augmented reality, CGI in Computer Graphics
  17. 17. Transformative motion
  18. 18. Transformative motion • ‘Shapes themselves are transformed gradually over time, thus changing their inherent nature’ (Woolman 2004) • Widely used in linear media • Related terms are ‘shape transformation’, ‘image morphing’ and ‘tweening’ (in-betweening), used in traditional animation, computer animation and motion graphics • A motion graphics textbook describes the process of transformation as either reductive, elaborative, or distortive (Woolman 2004)
  19. 19. Non-sequential navigation
  20. 20. Non-sequential navigation • Traditional hypertext systems are based on websites (or nodes) connected by links. The navigation may be non-linear, but is still sequential • If a user continuously affects the navigation, there are no longer any predefined nodes. Thus, the navigation is non-sequential. • Wayfinding in architecture (Lynch 1960) • Structural patterns in hypertext: cycle, counterpoint, mirrorworld, tangle, sieve, montage, split/join, missing link, feint (Bernstein 1998)
  21. 21. Screened spatialisation
  22. 22. Screened spatialisation • Motion is used to create, enhance, manipulate, or distort the sense of two or three-dimensional space as we know it • Interface becomes a virtual camera, moving in virtual space • Interface in physical space • Camera movement in film, such as zoom, pan, dolly (Zettl 1990) • Also used in electroacoustic music to denote sounds’ different sources in space or sounds’ spatial movement • Embodied interaction (Dourish 2004)
  23. 23. Reflections
  24. 24. Some implications for design & media education • textually, interfaces and navigation are becoming more dynamic as multimodal ‘compositions’ • in designing there is a need to know new computational features and communicative functions • a working vocabulary for navimation may be informed by design & through terms from developers • multimodal discourse that includes navimation may be built in and through action, via production with analysis • design and media students may be drawn together in collaborative production and reflection on navimation as part of their wider critical social semiotic analyses and emerging practices of mediated meaning making
  25. 25. Further research in navimation • RECORD: a multi-level research by design project • Phase 1: features, characteristics, concepts, consultations, ‘sketching’ • Phase 2: collaborative design, creativ eindustry parrtners and users • Phase 3: collaborative design with greater user iteration and evaluation • research publications, including an article based PhD
  26. 26. Design examples
  27. 27. References
  28. 28. Bernstein, M. (1998). ‘Patterns of hypertext’. Hypertext ‘98. ACM. Burn, Andrew & Parker, David. (2001). ’Making your mark: digital inscription, animation and a new visual semiotic’. Education, Communication and Information.Vol. 1, No. 2. Cope, B. & Kalantzis, M. 2000. (Eds). Multiliteracies: Literacy Learning and the Design of Social Futures. London: Routledge. Dourish, P. (2001). Embodied Interaction. The MIT Press: Cambridge MA. Fishwick, P. (2006). (Ed.). Aesthetic Computing. The MIT Press: Cambridge. Ivarsson, J. & Saljö, R. (2005). ‘Seeing through the screen: human reasoning and the development of representational technologies’. In Gärdenfors, P. & Johansson, P. (Eds). Cognition, Education and Communication Technology. Lawrence Erlbaum Associates: Mahwah. 203-222. Jewitt, C. & Kress, G. (2003). Multimodal Literacy. New York: Peter Lang. Jewitt, C. (2005). Technology, Literacy, Learning: A Multimodal Approach. Routledge London. Kress, G. (2003). Literacy in the New Media Age. London: Routledge. Kress, G. & van Leeuwen, T. (1996). Reading Images: the grammar of visual design. London: Routledge. Kress, G. & van Leeuwen, T. (2001). Multimodal Discourse: the Modes and Media of Contemporary Communication. London: Arnold. Lemke, J. (1998). ‘Metamedia literacy: transforming meanings and media.’ In Reinking, D.; McKenna, M., Labbo, L. & Kieffer, R.. (Eds). Handbook of Literacy and Technology: transformations in a post-typographic world. Mahwah: Lawrence Erlbaum. 283-301. Lemke, J. (2005). ‘Place, pace, and meaning: multimedia chronotopes’. In Norris, S. & Jones, R.. (Eds). Discourse in Action: Introducing Mediated Discourse Analysis. Routledge: London. 110-122. Löwgren, J. & Stolterman, E. (2004). Thoughtful Interaction Design. The MIT Press: Cambridge MA. Manovich, L. (2001). The Language of New Media. The MIT Press: Cambridge MA. Morrison, A. (2008 in press). Inside Multimodal Composition. Hampton Press: Cresskill. Shneiderman, B., Bederson, B. & Drucker, S. (2006). ‘Find that photo! Interface strategies to to annotate, browse and share’. Communications of the ACM. Vol. 49, no. 4. 69-71. Skjulstad, S. (2007). 'Motion graphics and communication design on the web'. Journal of Media Practice. Skjulstad, S & Morrison, A. (2005). ‘Movement in the interface’. Computers & Composition. Vol. 22, No. 4. 413-433. Stöckl, H. (2004). ‘In between modes: language and image in printed media’. In Ventola E. Charles, C. & Kaltenbacher, M. (Eds). Perspectives on Multimodality.  Amsterdam: John Benjamins. 9-30. Ulmer, G. (1994). Heuretics. The Johns Hopkins Press: Baltimore. Ulmer, G. (2003). Internet Invention: From Literacy to Electracy. Longman: New York. Wood, A. 2007. Digital Encounters. Routledge. London. Woolman, M. 2004. Motion Design: Moving Graphics for Television, Music Video, Cinema, and Digital Interfaces: RotoVision. Zettl, H. 1990. Sight, sound, motion, applied media aesthetics. 2nd ed. Belmont, CA: Wadsworth.