Talk "Multimodal Systems", VDMA ERFA Meeting, 05/2020

FROM MULTITOUCH TO
MULTIMODAL
HUMAN-CENTRED DIGITISATION
Bildquelle: https://ooc.vn/ung-dung-cua-big-data/
JAN GROENEFELD - SOLUTION MANAGER INDUSTRY | ERGOSIGN

PREFACE
The times of the "single point of interaction" on the wide format multi-
touch screen are numbered. Multiple touch points with cross-modality
interaction options - from touch to speech to gestures - will become
standard in the everyday work of machine operators in the medium
term. In addition, there is a progressive interconnectedness of our
physical environment, as well as the digital-sensory extension of the
human being himself.
What counts now is a human-centered view: on the one hand, in terms
of the change of the work situation, on the other hand, in terms of the
way our senses perceive and function. Which technologies have the
potential to support people in their work in a meaningful way in the
short and long term? Which methods and design principles are suitable
for designing multimodal systems in a way that they support people
instead of overburdening them?
Take a look with us at future work situations and application contexts.
We present human-centered process models based on service design,
which enable the successful development of smart industrial
applications and holistically interactive ecosystems in an industrial
context.
Jan Groenefeld
Solution Manager Industry
Lead UX Designer
jan.groenefeld@ergosign.de
T +49 681 95 98 20 0

CONTENTS
Short Introduction
Warm-up
Sense(s) & Nonsense
Multimodal = Multicomplex?
Design of Multimodal Systems
Conclusion and Outlook
ANNEX: Ergosign - Company Presentation

GOALS OF THIS TALK
Raising awareness and getting fit for the
”post-touch era“.
Understanding of human senses and their
consideration in the development of
interactive systems.
Holistic view of future human-machine
interaction with modern technologies.
A feeling for potential and danger.
Methodical approaches and
recommendations for the design of
multimodal systems through service design.

JAN GROENEFELD
SOLUTION MANAGER INDUSTRY,
LEAD UX DESIGNER - ERGOSIGN SAARBRÜCKEN
PROFILE
Jan Groenefeld has been Lead User Experience Designer and Solution
Manager for the Industry division at Ergosign GmbH in Saarbrücken since
2006. He studied Digital Media and, in addition to his work in an industrial
context, is enthusiastic about everything that has to do with consumer
gadgets, sports and autonomous, electrified mobility concepts.
The passion for modern technologies and their use, added value and
influences on the user 4.0 of the much-cited Smart Factory have
accompanied him throughout the past 14 years as a designer of human-
machine interfaces.
Consequently, his project focus is on innovative operating concepts in the
environment of industrial production plants, from the machine itself to
control stations, cross-company IIoT platforms and business intelligence
applications in the context of predictive maintenance and condition
monitoring.
My mission: Making modern technologies usable for people.
HI!
Bildquelle: Ergosign GmbH

ERGOSIGN
WE INNOVATE & CREATE
DIGITAL EXPERIENCES
We are pioneers in user experience design and are one
of the leading agencies in Europe. Our services include
strategic consulting, design and development. Benefit
from our experience and our joint commitment to the
digital transformation of your processes and products.
BEST OF
2018
2018

3200SUCCESSFUL PROJECTS
In the Enterprise, Industry,
Health and Consumer sectors
ERGOSIGN
ISO 9001 CERTIFIED QUALITY
182UX ENTHUSIASTS
At six locations in Berlin,
Hamburg, Munich, Saarbrücken,
Stuttgart and Zurich.
23%GROWTH
Average growth over the last 5
years with 96% customer
satisfaction

98%
UNIT
ENTERPRISE
INDUSTRY
HEALTH
CONSUMER
2018

"I think there's a worldwide
market for maybe ﬁve
computers."
Thomas Watson, CEO IBM, 1943
V
ISIO
N
A
RY
;)
Bildquelle: https://forbes.es/emprendedores/7457/10-lecciones-de-thomas-j-watson-primer-ceo-de-ibm/

“Internet is just a hype...”
Bill Gates, 1993
Bildquelle: https://www.computerworld.ch/business/microsoft/bill-gates-microsoft-verlassen-1333326.html
V
ISIO
N
A
RY
;)

https://www.it-production.com/news/maerkte-und-trends/microsoft-iot-signals-report/
30BILLIONNETWORKEDIOTDEVICES
UPTO75BILLIONUNTIL2025
Bildquelle: https://www.bosch-presse.de/pressportal/de/de/wandel-von-bosch-zum-iot-unternehmen-137474.html

NO-TOUCH
THE FIRST
COMPUTERS
were revolutionary, but almost
exclusively geared towards specialists.
https://space10.com/project/the-augmented-human/
Illustration — Marie Mohanna

MULTI-TOUCH
THE SMARTPHONE
REVOLUTION
…didn’t just make life more
convenient. It fundamentally changed
our relationship with technology by
making it immediate and close to our
bodies.

MULTIMODAL
+ MULTIMEDIAL
SMART GLASSES
…once technology moves so close to
our senses, something interesting
happens: we move from looking at the
interface to stepping into it, and with
that the mediating role of technology
has the potential to all but disappear
into the background.

UBIQUITOUS
MULTI-
EXPERIENCE
DEMISE OF THE
DEVICE
…and the rise of the accessory, we
will discover a world in which screens
and interfaces are both increasingly
closer to our bodies (thanks to
wearables and other interfaces) and all
around us (thanks to increasingly
immersive and augmented
environments).

"NOTHING IS AS
IT SEEMS!
Bildquelle: https://www.moviepilot.de/news/x-factor-das-unfassbare-so-unfassbar-kamen-die-neuen-folgen-bei-den-fans-an-1112913
”

98%
UNIT
Illusion of
Movement & Size
Quelle: https://www.blickcheck.de/auge/funktion/optische-taeuschungen/

PERCEPTION IS
HIGHLY INDIVIDUAL.
Bildquelle: https://www.oceando.de/kreuzfahrt-reiseziele/kreuzfahrtgebiet-suedsee.html
Silence
Recovery
Refreshing
Sunburn
Heat
Beach too "sandy"

USER EXPERIENCE ALSO
STARTS WITH OUR
SENSES.
Bildquelle: https://www.oceando.de/kreuzfahrt-reiseziele/kreuzfahrtgebiet-suedsee.html
INDIVIDUAL PREFERENCES
Interactions with devices are - depending on the individually
preferred modality (visual, auditory, ...) - perceived as positive or
exhausting.

OUR
SENSES
SEE
LISTEN
FEEL
TASTE
SMELL
RESTRICTIONS ON PERCEPTION
Color vision defects - 8% men in Germany
Visual impairment - approx. 2/3 of people
Hearing loss - approximately 1/5 of people
Regular temporary perception limitations even in healthy
people.
Bildquelle: https://www.kurier.de/inhalt.geschaerfte-sinne-sinnesorgane-mit-ueberdurchschnittlichen-faehigkeiten.2b2d38cd-2f71-4ef2-aa14-e183a596e0f8.html

HUMAN
MODALITIES
VISUAL
AUDITIVE
HAPTICAL
PROPRIOZEPTIVE
Bildquelle: https://markuscerenak.com/dankbarkeit-fuehlen.html
Bildquelle: http://shop.oreilly.com/product/0636920049500.do
Bildquelle: https://www.rhein-zeitung.de/ratgeber/fit-
und-gesund/gesundheitstipps_artikel,-faszination-
ohr-25000-haarzellen-auf-hochbetrieb-
_arid,2097516.html
Bildquelle: https://www.blickcheck.de/auge/funktion/
sehen/

MULTI-
MODALITIES
LIFE SUPPORT
ORIENTATION
HAND-EYE COORDINATION
SOCIAL INTERACTION
COGNITION AND ANALYSIS
Bildquelle: https://www.readersdigest.ca/health/healthy-living/dai-manuel-unhealthy-food-trends/
Bildquelle: https://thehungryjpeg.com/product/3483722-isometric-3d-vector-charts-set-for-infographics
Touch Interface per Definition
bereits multimodale Interaktion!
Bildquelle: https://sports.bwin.com/de/news/weitere-sportarten/tennis/us-open-favoriten-prognose-quoten-wetten

MECHANISMS OF PERCEPTION
RELEVANT TO DESIGN
FILTER
Ability to permanently filter out stimuli on each sensory channel in
order to protect us from being overwhelmed (e.g. "cocktail party effect"
- filtering out other persons).
SUBSTITUTION & TRANSLATION
Ability to spontaneously use alternative modalities or "translation of
environmental parameters" when selected senses are blocked (e.g.
interpreting the speed of a car through rolling noise).
SENSORICAL FOCUS & SHIFTS
Usually one activity is dominated by one modality. Changes of modality
within an activity cost a lot of attention and should be avoided.
SYSTEM 1 vs. SYSTEM 2
Learned modalities sometimes go by "without thinking" (system 1).
Newly learned modalities cost a lot of cognitive energy (system 2).
System 2 activities should only be used to a limited extent.
RHYTHM & REPETITION
help to build a "skill" from a new modality.

The goal of multimodal interaction is to make human-
computer interaction more "natural" than with
unimodal applications.
This often involves attempts to "imitate" the
interaction between people and real environments.
!

AUGMENTED HUMAN
Sensory, cognitive and physical "extension" of the
human being.
Bildquelle: https://www.geospatialworld.net/blogs/wearable-rape-sensor/

98%
UNIT
EXAMPLES OF
MULTIMODAL
INTERACTION
http://seoulspace.com/2016/04/07/samsung-gets-patent-for-ar-contact-lenses/
SAMSUNG PATENT (2019)
Bildquelle: https://www.slant.co/topics/6097/
~hololens-games
Bildquelle: https://www.inside-digital.de/tests/apple-
watch-series-5-im-test-ein-echtes-smartwatch-
highlight-mit-einem-grossen-aber
Bildquelle: https://zkm.de/de/event/2015/05/
malen-kann-jeder-aber-auf-dem-ipad
Bildquelle: https://mixed.de/ikea-digital-die-wohnung-
einrichten-update-fuer-ar-app/

DOMINANT FORM OF
COMMUNICATION IS
SPEAKING
ONLY SEEMINGLY EFFORTLESS
We have innate mechanisms that make complex language possible.
It only seems to come easily to us. This is due to the constant practice
and necessity for social interaction. In fact, speaking ties up many
resources.
Bildquelle: https://entrepreneurmindz.com/author/audrey/

Bildquelle: https://factorynet.at/a/wie-exoskelette-fuer-die-industrie-funktionieren
SUPER POWER
Thanks to exoskeletons, expansion of the human being happens on
cognitive and physical level.

BRAIN
CONTROL
INTERFACE
Bildquelle: https://nt.interia.pl/gadzety/news-robot-sterowany-mysla,nId,685098
A shortcut in the modality model past our senses -> New type of
modality?

DEVICE MODES
INPUT/OUTPUT
TONE/LANGUAGE
VISUALLY
HAPTIC
…
Device Mode
Bildquelle: http://shop.oreilly.com/product/0636920049500.do
Bildquelle: https://www.youtube.com/watch?v=Gdww2j76vvs
Monitor
Input
Analyze
Decide
Respond
& control
Output
Bildquelle: https://entrepreneurmindz.com/author/audrey/

Example
BMW i Interaction Ease
Context-sensitive interpretation
of multimodal interactions using
AI
Quelle: https://www.youtube.com/watch?v=Vi1mPl5gO8c

Bildquelle: https://www.wired.com/2011/02/giant-spectacular-multi-touch-screen-wraps-around-you/
PERSONALIZED &
INTERACTIVE
ENVIRONMENT

OLFACTORY
INTERFACES
Bildquelle:
https://odditymall.com/ophone-lets-you-send-
smells-digitally-to-your-friends
Bildquelle: https://www.cplaromas.com/fragrance-trends/tech-scent-olfactory-ideas-for-the-modern-world/
Bildquelle: https://thenextweb.com/shareables/2013/10/17/wake-smell-
virtual-coffee-ophone-will-take-breath-away/

Bildquelle: https://www.all-electronics.de/5-tipps-fuer-die-auswahl-mobiler-roboter/ Bildquelle: https://www.homag.com/news-events/news/artikel/roboter-technologie-bietet-chancen-fuer-den-
holzhausbau
Lernen
Laufen Greifen Sprechen Zuhören
Sehen
Lesen Analysieren
Smarte Maschinen können
Lernen
Sehen
Lesen Analysieren
Lernen
Sehen
Lesen Analysieren
Lernen
Sehen
Lesen Analysieren
Smarte Maschinen könnenSTATUS QUO
IN SMART FACTORY
Machines already mimic a variety of human sensory
capabilities and associated modalities.

Bildquelle: https://www.powertransmissionworld.com/egp-c-co-act-by-schunk-the-first-certified-industrial-gripper-for-collaborative-operation/
COBOTS
ISO 15066
Working in MRC systems (human-robot collaboration)
hand in hand with humans.
Bildquelle: https://www.fraunhofer.at/content/dam/austria/documents/WhitePaperTUEV/
White%20Paper_Sicherheit_MRK_Ausgabe%201.pdf

DAIMLER - FACTORY 56
MOST MODERN CAR PRODUCTION IN
EUROPE
300 driverless transport systems and CoBots
Use of AI for planning processes/predictive
maintenance and optimization of
manufacturing processes.
5G for latency-free and fail-safe
communication.
Flexible production lines and large quantities
with high product variety.
Bildquelle: https://www.automobilwoche.de/article/20200206/HEFTARCHIV/200209978/das-ende-des-
fliessbands-mit-der-factory--will-daimler-die-montage-revolutionieren

The irreversible technologization and complexity of our
private and working environment makes it
necessary ...
... to create multimodal systems in the future that
make all available senses (or modalities) usable for
eﬃcient interaction with the environment and
machines.
!

HOW TO
DESIGN
MULTIMODAL
SYSTEMS

SERVICE DESIGN+
CLASSIC UI/UX &
PRODUCT DESIGN
The key lies in the
extension and combination of methods.

SERVICE DESIGN UX DESIGN
Holistic
Multiple Touchpoints
Open
Broad Research
Value-based
Strategic
Visionary
Concrete
Defined Touchpoints
Focused Research
Requirement-based
Detail Design
UI Specification
Handover to Developers
Quelle: Ergosign GmbH

Service Design is user-centered, analytical, interdisciplinary and
provides useful products, processes and services.
Service designers look at the complete process of a service and
design it as smoothly and consistently as possible.
Service design does not only concern the design of interfaces
and user experiences, but also includes the processes and
organizational models within companies. It forms an interface
between business, marketing, design and technology and can
reduce silo thinking through a collaborative, holistic approach.
In our view, the customer journey (map) is a central instrument
for the holistic representation of the user experience (status
quo and in the future). On the following page you can see where
we typically use this method.
SERVICE DESIGN METHODS
E.G. CUSTOMER JOURNEYS

We carry out our research activities
where your customers make their
experiences. With our outside-in
perspective, we gain valuable insights and
identify opportunities to improve and
expand customer travel.
We create a holistic picture of the
experience your customers have with you,
your products and your brand. This holistic
picture has great potential to uncover
unfulfilled expectations and weaknesses in
the customer experience.
CURRENT CUSTOMER
EXPERIENCE
INTERNAL/EXTERNAL
INVESTIGATION
PREVIEW OF FUTURE
SERVICE
In a design strategy we show shared benefits for
your customers, your business and your
employees and how you can achieve them.
Insight-based recommendations for targeted
improvement of your services are merged with
existing strategies to create a visionary picture.
Stakeholder Interviews
Contextual Inquiries
Observations
Insight & Opportunity Report
Personas
Ecosystem Map
Customer Journeys/Experience Map
Blueprints
Customer Journeys (Future)
Design Vision
Value-driven Roadmap
SERVICE DESIGN

METHODS
Personas + Empathy Maps
User Story Mapping
Month in a Life
Behavioral Maps
Storyboards
https://www.pexels.com/photo/tool-set-on-plank-175039/
Journey Map
Context Map
Focus Map
Design Fiction
…
GOAL
Systematic collection, analysis and documentation of the
entire sensory perception in one process.
Conclusions on the optimal use of suitable modalities while
avoiding cognitive overload in subsequent operation.

MULTIMODAL
JOURNEY MAP
Extension of the classic journey map by dimensions
for the design of multimodal systems:
Focus Requirements:
focal, peripheral, interruptive, filtered modalities
Modality Characteristics:
sequence, continuity, interruptions/distractions
Multimodal User Behaviors:
sensing, understanding, deciding, acting
Multimodal Product Behaviors:
monitor, analyze, decide, respond, control
https://www.pexels.com/photo/tool-set-on-plank-175039/ Quelle: Ergosign GmbH

Context
Insights&
Opportunities
ModalityConditionsProcess&Experience
STAGE
ACTIVITIES &
LEADING MODALITIES
EMOTIONAL &
PHYSICAL STATES
SENSING
UNDERSTANDING/
DECIDING
ACTING
EXPERIENCE GOAL
WER, WANN, WARUM?
Leitstandbediener
im Stahlwerk
MULTIMODAL
JOURNEY MAP
(AUSZUG)
Fehlerfall an
Walzanlage -
Bramme aus Bahn
geschleudert
Ziel:
Personenschaden
verhindern
Ziel: Ausfallzeit
der Anlage gering
halten
…
Fehlfunktion in
der Anlage…
1. Alarm wird von
Nutzer registriert
Stress, Angst
“Was war das?
“Hoffentlich ist
dem Kollegen
nichts passiert”
Auditiv, Visuell
Hören & Sehen
von Meldungen im
System
Automatische
Rückmeldung zu
Koll.-Gesundheit
“Dimmen” des
Alarms nach
Bestätigung
“Pre-Alert” als
Vorstufe
Augen und Ohren
zu großen Teilen
blockiert
Lärm & Hektik
erschweren
Fehlerortung
Aus
Stresssituation
herauskommen
…
2. Orientierung
& Behebung
Visuell, Propriozeptiv
“Niemand zu
Schaden
gekommen”
Erste
Statusanalyse
mittels SCADA
“Was mache ich
zuerst ?”
Neustart der
Anlage
Innere
Anspannung löst
sich langsam…
Sicht auf Anlage
erschwert durch
Rauch
…
Rücksprache mit
Kollegen im Feld
Soziale Interakt.
Lage in den Griff
bekommen
Anlagenneustart ist
in Stresssituation
kompliziert
Nutzer benötigt
Führung…
Sensorische
Sichterweiterung durch
autonome Drohne im Feld,
Steuerung per Smartglasses.
Bestätigung der einzelnen
Schritte zum geführten
Neustart der Anlage per
Spracheingabe
Only the holistic understanding of emotions, senses
and dominant modalities enables informed design
decisions in the design process
of multimodal systems and the targeted use of modern
(interaction) technologies.
!

DESIGN PRINCIPLES FOR
MULTIMODAL SYSTEMS
Ensure sensory focus (avoid jumps).
Limit "cognitive load" (limited number of
processable input sources).
Ensure redundancy and reliability (substitution
and translation of modalities in case of
temporary sensory impairment).
Avoid or specifically design shifts, interruptions
and flows (see systems 1 & 2).
System feedback (output) on different sensory
channels to facilitate validation .
Use "Body Language" also for interactive
systems (see MRKs ISO 15066).
Bildquelle: https://nonprofitmegaphone.com/ultimate-list-50-nonprofit-marketing-ideas/

WELL-INTENTIONED
IS NOT ALWAYS
WELL DONE
Bildquelle: https://www.otz.de/leben/vermischtes/auto-
verkehr/der-countdown-laeuft-thueringen-will-neue-ampeln-
fuer-fussgaenger-testen-id220342223.html
Bildquelle: https://www.audicup.com/de/experience-audi/mobility-and-trends/urban-life/smart-city.html
Additional information (timer) is intended to simplify the assessment of the
situation when crossing the road.
However, studies suggest that the additional information leads to a
disproportionate use of cognitive resources.
In an environment of any complexity, such as a "Smart City", this could
make it unnecessarily difficult to detect other situations.

Bildquelle: https://www.tagesspiegel.de/wirtschaft/virtual-reality-die-360-grad-computerwelt/12790976.html
Bildquelle: https://www.azeron.eu
NEW MODALITIES
OFTEN REQUIRE A
LEARNING PHASE
The adaptation or variation of existing modalities
is not always effortless.
At the same time, it is necessary to wait for the
adaptation phase to be completed before drawing
conclusions about performance.
Gamers with special input devices are often clearly
superior to their opponents in a short time.

MR.
MULTIMODAL
Bildquelle: https://www.express.co.uk/life-style/science-technology/576707/Minority-Report-Gloves-Military-General-BAE-Systems

DESIGN OF
MULTIMODAL SYSTEMS
(MMS) - TAKE AWAYS
UX starts with our senses. A holistic understanding
of this in usage contexts is indispensable for the
design of MMS.
Unexplored, therefore research and flexible reaction
to findings in the design process are important.
Great potential of unused modalities. But: an
apparently suitable modality is not always perceived
as "better" (regular user feedback indispensable).
Post-touch era unavoidable in order to remain on par
with the highly engineered environment of the near
future.
Appropriate design of MMS can only be realized with
the help of an extended method set consisting of
service design and product design.
!

WHAAAT?
Bildquelle: https://thesmartlocal.com/read/unknown-pokka-drinks/
CONTACT ME: JAN.GROENEFELD@ERGOSIGN.DE

THANKS A
LOT!
Jan Groenefeld
Solution Manager Industry
Lead UX Designer
jan.groenefeld@ergosign.de
T +49 681 95 98 20 0
BOOK RECOMMENTATIONS ON THE TOPIC

We innovate
and create
digital experiences.

We believe in digital products that
empower people in their work and
delight their everyday lives.

Our full-service approach is based on
a human-centered, goal-oriented and
collaborative mindset.

Meet us in our 7 oﬃces in
Germany, Switzerland and
China for an inspiring chat.

We are 170+ friendly,
open-minded and
award-winning

We are passionate
about exploring future
trends and drive digital
transformation.

We wrote a
book about
collaborative
methods in
UX Design.

We serve our customers from a wide
variety of industries and business areas,
both B2B and B2C.

ERGOSIGN SHOWREEL
CLICK TO WATCH
https://res.cloudinary.com/ergosign/video/upload/vc_h265,w_1280/videos/UI_Reel_full_201912.mp4

Let’s shape the
future together!

Talk "Multimodal Systems", VDMA ERFA Meeting, 05/2020

Recommended

Recommended

More Related Content

Similar to Talk "Multimodal Systems", VDMA ERFA Meeting, 05/2020

Similar to Talk "Multimodal Systems", VDMA ERFA Meeting, 05/2020 (20)

More from Ergosign GmbH

More from Ergosign GmbH (20)

Recently uploaded

Recently uploaded (20)

Talk "Multimodal Systems", VDMA ERFA Meeting, 05/2020