ENG_From Multitouch to Multimodal - Design of multimodal systems with service design approach - Jan Groenefeld - Ergosign
•
0 likes•143 views
The times of the "single point of interaction" on the wide format multi- touch screen are numbered. Multiple touch points with cross-modality interaction options - from touch to speech to gestures - will become standard in the everyday work of machine operators in the medium term. In addition, there is a progressive interconnectedness of our physical environment, as well as the digital-sensory extension of the human being himself. What counts now is a human-centered view: on the one hand, in terms of the change of the work situation, on the other hand, in terms of the way our senses perceive and function. Which technologies have the potential to support people in their work in a meaningful way in the short and long term? Which methods and design principles are suitable for designing multimodal systems in a way that they support people instead of overburdening them? Take a look with us at future work situations and application contexts. We present human-centered process models based on service design, which enable the successful development of smart industrial applications and holistically interactive ecosystems in an industrial context.
Recommended
More Related Content
What's hot
What's hot (14)
Similar to ENG_From Multitouch to Multimodal - Design of multimodal systems with service design approach - Jan Groenefeld - Ergosign
Similar to ENG_From Multitouch to Multimodal - Design of multimodal systems with service design approach - Jan Groenefeld - Ergosign (20)
More from Jan Groenefeld
More from Jan Groenefeld (6)
Recently uploaded
Recently uploaded (20)
ENG_From Multitouch to Multimodal - Design of multimodal systems with service design approach - Jan Groenefeld - Ergosign
- 1. FROM MULTITOUCH TO MULTIMODAL HUMAN-CENTRED DIGITISATION Bildquelle: https://ooc.vn/ung-dung-cua-big-data/ JAN GROENEFELD - SOLUTION MANAGER INDUSTRY | ERGOSIGN
- 2. PREFACE The times of the "single point of interaction" on the wide format multi- touch screen are numbered. Multiple touch points with cross-modality interaction options - from touch to speech to gestures - will become standard in the everyday work of machine operators in the medium term. In addition, there is a progressive interconnectedness of our physical environment, as well as the digital-sensory extension of the human being himself. What counts now is a human-centered view: on the one hand, in terms of the change of the work situation, on the other hand, in terms of the way our senses perceive and function. Which technologies have the potential to support people in their work in a meaningful way in the short and long term? Which methods and design principles are suitable for designing multimodal systems in a way that they support people instead of overburdening them? Take a look with us at future work situations and application contexts. We present human-centered process models based on service design, which enable the successful development of smart industrial applications and holistically interactive ecosystems in an industrial context. Bildquelle: https://ooc.vn/ung-dung-cua-big-data/ Jan Groenefeld Solution Manager Industry Lead UX Designer jan.groenefeld@ergosign.de T +49 681 95 98 20 0
- 3. CONTENTS Short Introduction Warm-up Sense(s) & Nonsense Multimodal = Multicomplex? Design of Multimodal Systems Conclusion and Outlook ANNEX: Ergosign - Company Presentation Bildquelle: https://ooc.vn/ung-dung-cua-big-data/
- 4. GOALS OF THIS TALK Raising awareness and getting fit for the ”post-touch era“. Understanding of human senses and their consideration in the development of interactive systems. Holistic view of future human-machine interaction with modern technologies. A feeling for potential and danger. Methodical approaches and recommendations for the design of multimodal systems through service design. Bildquelle: https://ooc.vn/ung-dung-cua-big-data/
- 5. INTRO
- 6. JAN GROENEFELD SOLUTION MANAGER INDUSTRY, LEAD UX DESIGNER - ERGOSIGN SAARBRÜCKEN PROFILE Jan Groenefeld has been Lead User Experience Designer and Solution Manager for the Industry division at Ergosign GmbH in Saarbrücken since 2006. He studied Digital Media and, in addition to his work in an industrial context, is enthusiastic about everything that has to do with consumer gadgets, sports and autonomous, electrified mobility concepts. The passion for modern technologies and their use, added value and influences on the user 4.0 of the much-cited Smart Factory have accompanied him throughout the past 14 years as a designer of human- machine interfaces. Consequently, his project focus is on innovative operating concepts in the environment of industrial production plants, from the machine itself to control stations, cross-company IIoT platforms and business intelligence applications in the context of predictive maintenance and condition monitoring. My mission: Making modern technologies usable for people. HI! Bildquelle: Ergosign GmbH
- 7. ERGOSIGN WE INNOVATE & CREATE DIGITAL EXPERIENCES We are pioneers in user experience design and are one of the leading agencies in Europe. Our services include strategic consulting, design and development. Benefit from our experience and our joint commitment to the digital transformation of your processes and products. BEST OF 2018 2018 Bildquelle: Ergosign GmbH
- 8. 3200SUCCESSFUL PROJECTS In the Enterprise, Industry, Health and Consumer sectors ERGOSIGN ISO 9001 CERTIFIED QUALITY 182UX ENTHUSIASTS At six locations in Berlin, Hamburg, Munich, Saarbrücken, Stuttgart and Zurich. 23%GROWTH Average growth over the last 5 years with 96% customer satisfaction
- 10. WARM UP
- 11. "I think there's a worldwide market for maybe five computers." Thomas Watson, CEO IBM, 1943 V ISIO N A RY ;) Bildquelle: https://forbes.es/emprendedores/7457/10-lecciones-de-thomas-j-watson-primer-ceo-de-ibm/
- 12. “Internet is just a hype...” Bill Gates, 1993 Bildquelle: https://www.computerworld.ch/business/microsoft/bill-gates-microsoft-verlassen-1333326.html V ISIO N A RY ;)
- 14. NO-TOUCH THE FIRST COMPUTERS were revolutionary, but almost exclusively geared towards specialists. https://space10.com/project/the-augmented-human/ Illustration — Marie Mohanna
- 15. MULTI-TOUCH THE SMARTPHONE REVOLUTION …didn’t just make life more convenient. It fundamentally changed our relationship with technology by making it immediate and close to our bodies. https://space10.com/project/the-augmented-human/ Illustration — Marie Mohanna
- 16. MULTIMODAL + MULTIMEDIAL SMART GLASSES …once technology moves so close to our senses, something interesting happens: we move from looking at the interface to stepping into it, and with that the mediating role of technology has the potential to all but disappear into the background. https://space10.com/project/the-augmented-human/ Illustration — Marie Mohanna
- 17. UBIQUITOUS MULTI- EXPERIENCE DEMISE OF THE DEVICE …and the rise of the accessory, we will discover a world in which screens and interfaces are both increasingly closer to our bodies (thanks to wearables and other interfaces) and all around us (thanks to increasingly immersive and augmented environments). https://space10.com/project/the-augmented-human/ Illustration — Marie Mohanna
- 19. "NOTHING IS AS IT SEEMS! Bildquelle: https://www.moviepilot.de/news/x-factor-das-unfassbare-so-unfassbar-kamen-die-neuen-folgen-bei-den-fans-an-1112913 ”
- 20. 98% UNIT Illusion of Movement & Size Quelle: https://www.blickcheck.de/auge/funktion/optische-taeuschungen/
- 21. PERCEPTION IS HIGHLY INDIVIDUAL. Bildquelle: https://www.oceando.de/kreuzfahrt-reiseziele/kreuzfahrtgebiet-suedsee.html Silence Recovery Refreshing Sunburn Heat Beach too "sandy"
- 22. USER EXPERIENCE ALSO STARTS WITH OUR SENSES. Bildquelle: https://www.oceando.de/kreuzfahrt-reiseziele/kreuzfahrtgebiet-suedsee.html INDIVIDUAL PREFERENCES Interactions with devices are - depending on the individually preferred modality (visual, auditory, ...) - perceived as positive or exhausting.
- 23. OUR SENSES SEE LISTEN FEEL TASTE SMELL RESTRICTIONS ON PERCEPTION Color vision defects - 8% men in Germany Visual impairment - approx. 2/3 of people Hearing loss - approximately 1/5 of people Regular temporary perception limitations even in healthy people. Bildquelle: https://www.kurier.de/inhalt.geschaerfte-sinne-sinnesorgane-mit-ueberdurchschnittlichen-faehigkeiten.2b2d38cd-2f71-4ef2-aa14-e183a596e0f8.html
- 24. HUMAN MODALITIES VISUAL AUDITIVE HAPTICAL PROPRIOZEPTIVE Bildquelle: https://markuscerenak.com/dankbarkeit-fuehlen.html Bildquelle: http://shop.oreilly.com/product/0636920049500.do Bildquelle: https://www.rhein-zeitung.de/ratgeber/fit- und-gesund/gesundheitstipps_artikel,-faszination- ohr-25000-haarzellen-auf-hochbetrieb- _arid,2097516.html Bildquelle: https://www.blickcheck.de/auge/funktion/ sehen/
- 25. MULTI- MODALITIES LIFE SUPPORT ORIENTATION HAND-EYE COORDINATION SOCIAL INTERACTION COGNITION AND ANALYSIS Bildquelle: https://www.readersdigest.ca/health/healthy-living/dai-manuel-unhealthy-food-trends/ Bildquelle: https://thehungryjpeg.com/product/3483722-isometric-3d-vector-charts-set-for-infographics Touch Interface per Definition bereits multimodale Interaktion! Bildquelle: https://sports.bwin.com/de/news/weitere-sportarten/tennis/us-open-favoriten-prognose-quoten-wetten
- 26. MECHANISMS OF PERCEPTION RELEVANT TO DESIGN FILTER Ability to permanently filter out stimuli on each sensory channel in order to protect us from being overwhelmed (e.g. "cocktail party effect" - filtering out other persons). SUBSTITUTION & TRANSLATION Ability to spontaneously use alternative modalities or "translation of environmental parameters" when selected senses are blocked (e.g. interpreting the speed of a car through rolling noise). SENSORICAL FOCUS & SHIFTS Usually one activity is dominated by one modality. Changes of modality within an activity cost a lot of attention and should be avoided. SYSTEM 1 vs. SYSTEM 2 Learned modalities sometimes go by "without thinking" (system 1). Newly learned modalities cost a lot of cognitive energy (system 2). System 2 activities should only be used to a limited extent. RHYTHM & REPETITION help to build a "skill" from a new modality. Bildquelle: https://ooc.vn/ung-dung-cua-big-data/
- 28. The goal of multimodal interaction is to make human- computer interaction more "natural" than with unimodal applications. This often involves attempts to "imitate" the interaction between people and real environments. !
- 29. AUGMENTED HUMAN Sensory, cognitive and physical "extension" of the human being. Bildquelle: https://www.geospatialworld.net/blogs/wearable-rape-sensor/
- 30. 98% UNIT EXAMPLES OF MULTIMODAL INTERACTION http://seoulspace.com/2016/04/07/samsung-gets-patent-for-ar-contact-lenses/ SAMSUNG PATENT (2019) Bildquelle: https://www.slant.co/topics/6097/ ~hololens-games Bildquelle: https://www.inside-digital.de/tests/apple- watch-series-5-im-test-ein-echtes-smartwatch- highlight-mit-einem-grossen-aber Bildquelle: https://zkm.de/de/event/2015/05/ malen-kann-jeder-aber-auf-dem-ipad Bildquelle: https://mixed.de/ikea-digital-die-wohnung- einrichten-update-fuer-ar-app/
- 31. DOMINANT FORM OF COMMUNICATION IS SPEAKING ONLY SEEMINGLY EFFORTLESS We have innate mechanisms that make complex language possible. It only seems to come easily to us. This is due to the constant practice and necessity for social interaction. In fact, speaking ties up many resources. Bildquelle: https://entrepreneurmindz.com/author/audrey/
- 32. Bildquelle: https://factorynet.at/a/wie-exoskelette-fuer-die-industrie-funktionieren SUPER POWER Thanks to exoskeletons, expansion of the human being happens on cognitive and physical level.
- 33. BRAIN CONTROL INTERFACE Bildquelle: https://nt.interia.pl/gadzety/news-robot-sterowany-mysla,nId,685098 A shortcut in the modality model past our senses -> New type of modality?
- 34. DEVICE MODES INPUT/OUTPUT TONE/LANGUAGE VISUALLY HAPTIC … Device Mode Bildquelle: http://shop.oreilly.com/product/0636920049500.do Bildquelle: https://www.youtube.com/watch?v=Gdww2j76vvs Monitor Input Analyze Decide Respond & control Output Bildquelle: https://entrepreneurmindz.com/author/audrey/
- 35. Example BMW i Interaction Ease Context-sensitive interpretation of multimodal interactions using AI Quelle: https://www.youtube.com/watch?v=Vi1mPl5gO8c
- 38. Bildquelle: https://www.all-electronics.de/5-tipps-fuer-die-auswahl-mobiler-roboter/ Bildquelle: https://www.homag.com/news-events/news/artikel/roboter-technologie-bietet-chancen-fuer-den- holzhausbau Lernen Laufen Greifen Sprechen Zuhören Sehen Lesen Analysieren Smarte Maschinen können Lernen Laufen Greifen Sprechen Zuhören Sehen Lesen Analysieren Smarte Maschinen können Lernen Laufen Greifen Sprechen Zuhören Sehen Lesen Analysieren Smarte Maschinen können Lernen Laufen Greifen Sprechen Zuhören Sehen Lesen Analysieren Smarte Maschinen könnenSTATUS QUO IN SMART FACTORY Machines already mimic a variety of human sensory capabilities and associated modalities.
- 39. Bildquelle: https://www.powertransmissionworld.com/egp-c-co-act-by-schunk-the-first-certified-industrial-gripper-for-collaborative-operation/ COBOTS ISO 15066 Working in MRC systems (human-robot collaboration) hand in hand with humans. Bildquelle: https://www.fraunhofer.at/content/dam/austria/documents/WhitePaperTUEV/ White%20Paper_Sicherheit_MRK_Ausgabe%201.pdf
- 40. DAIMLER - FACTORY 56 MOST MODERN CAR PRODUCTION IN EUROPE 300 driverless transport systems and CoBots Use of AI for planning processes/predictive maintenance and optimization of manufacturing processes. 5G for latency-free and fail-safe communication. Flexible production lines and large quantities with high product variety. Bildquelle: https://www.automobilwoche.de/article/20200206/HEFTARCHIV/200209978/das-ende-des- fliessbands-mit-der-factory--will-daimler-die-montage-revolutionieren
- 41. The irreversible technologization and complexity of our private and working environment makes it necessary ... ... to create multimodal systems in the future that make all available senses (or modalities) usable for efficient interaction with the environment and machines. !
- 43. SERVICE DESIGN+ CLASSIC UI/UX & PRODUCT DESIGN The key lies in the extension and combination of methods. Bildquelle: Ergosign GmbH
- 44. SERVICE DESIGN UX DESIGN Holistic Multiple Touchpoints Open Broad Research Value-based Strategic Visionary Concrete Defined Touchpoints Focused Research Requirement-based Detail Design UI Specification Handover to Developers Quelle: Ergosign GmbH
- 45. Service Design is user-centered, analytical, interdisciplinary and provides useful products, processes and services. Service designers look at the complete process of a service and design it as smoothly and consistently as possible. Service design does not only concern the design of interfaces and user experiences, but also includes the processes and organizational models within companies. It forms an interface between business, marketing, design and technology and can reduce silo thinking through a collaborative, holistic approach. In our view, the customer journey (map) is a central instrument for the holistic representation of the user experience (status quo and in the future). On the following page you can see where we typically use this method. SERVICE DESIGN METHODS E.G. CUSTOMER JOURNEYS Quelle: Ergosign GmbH
- 46. We carry out our research activities where your customers make their experiences. With our outside-in perspective, we gain valuable insights and identify opportunities to improve and expand customer travel. We create a holistic picture of the experience your customers have with you, your products and your brand. This holistic picture has great potential to uncover unfulfilled expectations and weaknesses in the customer experience. CURRENT CUSTOMER EXPERIENCE INTERNAL/EXTERNAL INVESTIGATION PREVIEW OF FUTURE SERVICE In a design strategy we show shared benefits for your customers, your business and your employees and how you can achieve them. Insight-based recommendations for targeted improvement of your services are merged with existing strategies to create a visionary picture. Stakeholder Interviews Contextual Inquiries Observations Insight & Opportunity Report Personas Ecosystem Map Customer Journeys/Experience Map Blueprints Customer Journeys (Future) Design Vision Value-driven Roadmap SERVICE DESIGN Quelle: Ergosign GmbH
- 47. METHODS Personas + Empathy Maps User Story Mapping Month in a Life Behavioral Maps Storyboards https://www.pexels.com/photo/tool-set-on-plank-175039/ Journey Map Context Map Focus Map Design Fiction … GOAL Systematic collection, analysis and documentation of the entire sensory perception in one process. Conclusions on the optimal use of suitable modalities while avoiding cognitive overload in subsequent operation. Quelle: Ergosign GmbH
- 48. MULTIMODAL JOURNEY MAP Extension of the classic journey map by dimensions for the design of multimodal systems: Focus Requirements: focal, peripheral, interruptive, filtered modalities Modality Characteristics: sequence, continuity, interruptions/distractions Multimodal User Behaviors: sensing, understanding, deciding, acting Multimodal Product Behaviors: monitor, analyze, decide, respond, control https://www.pexels.com/photo/tool-set-on-plank-175039/ Quelle: Ergosign GmbH
- 49. Context Insights& Opportunities ModalityConditionsProcess&Experience STAGE ACTIVITIES & LEADING MODALITIES EMOTIONAL & PHYSICAL STATES SENSING UNDERSTANDING/ DECIDING ACTING EXPERIENCE GOAL WER, WANN, WARUM? Leitstandbediener im Stahlwerk MULTIMODAL JOURNEY MAP (AUSZUG) Fehlerfall an Walzanlage - Bramme aus Bahn geschleudert Ziel: Personenschaden verhindern Ziel: Ausfallzeit der Anlage gering halten … Fehlfunktion in der Anlage… 1. Alarm wird von Nutzer registriert Stress, Angst “Was war das? “Hoffentlich ist dem Kollegen nichts passiert” Auditiv, Visuell Hören & Sehen von Meldungen im System Automatische Rückmeldung zu Koll.-Gesundheit “Dimmen” des Alarms nach Bestätigung “Pre-Alert” als Vorstufe Augen und Ohren zu großen Teilen blockiert Lärm & Hektik erschweren Fehlerortung Aus Stresssituation herauskommen … 2. Orientierung & Behebung Visuell, Propriozeptiv “Niemand zu Schaden gekommen” Erste Statusanalyse mittels SCADA “Was mache ich zuerst ?” Neustart der Anlage Innere Anspannung löst sich langsam… Sicht auf Anlage erschwert durch Rauch … Rücksprache mit Kollegen im Feld Soziale Interakt. Lage in den Griff bekommen Anlagenneustart ist in Stresssituation kompliziert Nutzer benötigt Führung… Sensorische Sichterweiterung durch autonome Drohne im Feld, Steuerung per Smartglasses. Bestätigung der einzelnen Schritte zum geführten Neustart der Anlage per Spracheingabe Only the holistic understanding of emotions, senses and dominant modalities enables informed design decisions in the design process of multimodal systems and the targeted use of modern (interaction) technologies. ! Quelle: Ergosign GmbH
- 50. DESIGN PRINCIPLES FOR MULTIMODAL SYSTEMS Ensure sensory focus (avoid jumps). Limit "cognitive load" (limited number of processable input sources). Ensure redundancy and reliability (substitution and translation of modalities in case of temporary sensory impairment). Avoid or specifically design shifts, interruptions and flows (see systems 1 & 2). System feedback (output) on different sensory channels to facilitate validation . Use "Body Language" also for interactive systems (see MRKs ISO 15066). Bildquelle: https://nonprofitmegaphone.com/ultimate-list-50-nonprofit-marketing-ideas/
- 51. WELL-INTENTIONED IS NOT ALWAYS WELL DONE Bildquelle: https://www.otz.de/leben/vermischtes/auto- verkehr/der-countdown-laeuft-thueringen-will-neue-ampeln- fuer-fussgaenger-testen-id220342223.html Bildquelle: https://www.audicup.com/de/experience-audi/mobility-and-trends/urban-life/smart-city.html Additional information (timer) is intended to simplify the assessment of the situation when crossing the road. However, studies suggest that the additional information leads to a disproportionate use of cognitive resources. In an environment of any complexity, such as a "Smart City", this could make it unnecessarily difficult to detect other situations.
- 52. Bildquelle: https://www.tagesspiegel.de/wirtschaft/virtual-reality-die-360-grad-computerwelt/12790976.html Bildquelle: https://www.azeron.eu NEW MODALITIES OFTEN REQUIRE A LEARNING PHASE The adaptation or variation of existing modalities is not always effortless. At the same time, it is necessary to wait for the adaptation phase to be completed before drawing conclusions about performance. Gamers with special input devices are often clearly superior to their opponents in a short time.
- 55. DESIGN OF MULTIMODAL SYSTEMS (MMS) - TAKE AWAYS UX starts with our senses. A holistic understanding of this in usage contexts is indispensable for the design of MMS. Unexplored, therefore research and flexible reaction to findings in the design process are important. Great potential of unused modalities. But: an apparently suitable modality is not always perceived as "better" (regular user feedback indispensable). Post-touch era unavoidable in order to remain on par with the highly engineered environment of the near future. Appropriate design of MMS can only be realized with the help of an extended method set consisting of service design and product design. Bildquelle: https://ooc.vn/ung-dung-cua-big-data/ !
- 56. WHAAAT? Bildquelle: https://thesmartlocal.com/read/unknown-pokka-drinks/ CONTACT ME: JAN.GROENEFELD@ERGOSIGN.DE
- 57. THANKS A LOT! Jan Groenefeld Solution Manager Industry Lead UX Designer jan.groenefeld@ergosign.de T +49 681 95 98 20 0 BOOK RECOMMENTATIONS ON THE TOPIC
- 59. We innovate and create digital experiences.
- 60. We believe in digital products that empower people in their work and delight their everyday lives.
- 61. Our full-service approach is based on a human-centered, goal-oriented and collaborative mindset.
- 63. Meet us in our 7 offices in Germany, Switzerland and China for an inspiring chat.
- 64. We are 170+ friendly, open-minded and award-winning
- 65. We are passionate about exploring future trends and drive digital transformation.
- 66. We wrote a book about collaborative methods in UX Design.
- 67. We serve our customers from a wide variety of industries and business areas, both B2B and B2C.
- 68. ERGOSIGN SHOWREEL CLICK TO WATCH https://res.cloudinary.com/ergosign/video/upload/vc_h265,w_1280/videos/UI_Reel_full_201912.mp4
- 69. Let’s shape the future together!