This document outlines the agenda for a workshop on hybrid intelligence. The agenda includes an introduction to hybrid intelligence, three workshop sessions to discuss related topics, presentations on applications and challenges of hybrid intelligence, and a closing session. Coffee breaks are scheduled between sessions. The goal is to bring together researchers to discuss the state of the art in hybrid intelligence and develop a research agenda on issues around the interplay of humans and machines.

1. The Interplay of Machines and Humans:
State of the Art and a Research Agenda for
Hybrid Intelligence

2. Agenda
Time Top
09:00 – 09:30 Get-together and Introduction
09:30 – 09:45 A short intro to Hybrid Intelligence
09:45 – 10:30 Workshop Session 1
10:30 – 10:50 Coffee Break
11:00 – 12:30 Workshop Session 2
12:30 – 13:00 Hybrid Intelligence in Practice
13:00 – 14:00 Lunch Break
14:00 – 15:15 Workshop Session 3
15.15 – 15:30 Closing
15:30 – 16:00 Coffee/Tea

3. Agenda
Time Top
09:00 – 09:30 Get-together and Introduction
09:30 – 09:45 A short intro to Hybrid Intelligence
09:45 – 10:30 Workshop Session 1
10:30 – 10:50 Coffee Break
11:00 – 12:30 Workshop Session 2
12:30 – 13:00 Hybrid Intelligence in Practice
13:00 – 14:00 Lunch Break
14:00 – 15:15 Workshop Session 3
15.15 – 15:30 Closing
15:30 – 16:00 Coffee/Tea

4. Before we start…
… we should get to know each other

5. Please tell us about…
… who you are
… your background
… your research

6. Agenda
Time Top
09:00 – 09:30 Get-together and Introduction
09:30 – 09:45 A short intro to Hybrid Intelligence
09:45 – 10:30 Workshop Session 1
10:30 – 10:50 Coffee Break
11:00 – 12:30 Workshop Session 2
12:30 – 13:00 Hybrid Intelligence in Practice
13:00 – 14:00 Lunch Break
14:00 – 15:15 Workshop Session 3
15.15 – 15:30 Closing
15:30 – 16:00 Coffee/Tea

7. Hybrid Intelligence

8. Artificial Intelligence
DEFINITION
“Human intelligence
exhibited by machines”

9. SOME
EXAMPLES
OF AI

10. Machine Learning
DEFINITION
“A computer program is said to learn from
experience E with respect to some class of tasks T
and performance measure P if its performance at
tasks in T, as measured by P, improves with
experience E”

11. No Supervision is Dangerous…

12. No Supervision is Dangerous…

13. No Supervision is Dangerous…

14. HUMAN INTELLIGENCE
MACHINE INTELLIGENCE
Consistency
Speed and
Efficiency
Pattern
Recognition
Hard
Information
and
Analytical
Flexibility and
Transfer
Creativity and
Empathy
Various Data
Types and
Intuition
Bias and
Heuristics

15. HYBRID INTELLIGENCE SYSTEMS

16. Hybrid Intelligence
DEFINITION
“the ability to accomplish complex goals by
combining human and artificial intelligence to
collectively achieve superior results and
continuously improve by learning from each other..”
(Dellermann et al. forthcoming)

17. Tasks for Humans Augmenting AI
USING HUMANS IN THE LOOP
Feature Generation

18. Tasks for Humans Augmenting AI
USING HUMANS IN THE LOOP
Feature Generation
Teaching
Is this a lie?

19. Tasks for Humans Augmenting AI
USING HUMANS IN THE LOOP
Feature Generation
Teaching
Troubleshooting

20. OUR
LEARNING
SO FAR…
KNOWLEDGE
EXTRACTION AND
AGGREGATION
INCENTIVES AND
BUSINESS
INTEGRATION
INTERPRETABILITY/
TRANSPARENCY
TRUST

21. Agenda
Time Top
09:00 – 09:30 Get-together and Introduction
09:30 – 09:45 A short intro to Hybrid Intelligence
09:45 – 10:30 Workshop Session 1
10:30 – 10:50 Coffee Break
11:00 – 12:30 Workshop Session 2
12:30 – 13:00 Hybrid Intelligence in Practice
13:00 – 14:00 Lunch Break
14:00 – 15:15 Workshop Session 3
15.15 – 15:30 Closing
15:30 – 16:00 Coffee/Tea

22. ANALYSING CULTURAL INFLUENCE IN THE
USAGE OF RECOMMENDATION AGENTS

23. Agenda
Time Top
09:00 – 09:30 Get-together and Introduction
09:30 – 09:45 A short intro to Hybrid Intelligence
09:45 – 10:30 Workshop Session 1
10:30 – 10:50 Coffee Break
11:00 – 12:30 Workshop Session 2
12:30 – 13:00 Hybrid Intelligence in Practice
13:00 – 14:00 Lunch Break
14:00 – 15:15 Workshop Session 3
15.15 – 15:30 Closing
15:30 – 16:00 Coffee/Tea

24. Platzhalter Vortrag 2
Break – 30min

25. Agenda
Time Top
09:00 – 09:30 Get-together and Introduction
09:30 – 09:45 A short intro to Hybrid Intelligence
09:45 – 10:30 Workshop Session 1
10:30 – 10:50 Coffee Break
11:00 – 12:30 Workshop Session 2
12:30 – 13:00 Hybrid Intelligence in Practice
13:00 – 14:00 Lunch Break
14:00 – 15:15 Workshop Session 3
15.15 – 15:30 Closing
15:30 – 16:00 Coffee/Tea

26. INTERVENING INTERACTION - AN APPROACH
FOR THE EVOLUTION OF HIGHLY EFFICIENT
HYBRID SYSTEMS

27. MEASURING SOCIAL TOLERANCE AND SOCIAL
ACCEPTABILITY IN HYBRID TEAM
COLLABORATION

28. Agenda
Time Top
09:00 – 09:30 Get-together and Introduction
09:30 – 09:45 A short intro to Hybrid Intelligence
09:45 – 10:30 Workshop Session 1
10:30 – 10:50 Coffee Break
11:00 – 12:30 Workshop Session 2
12:30 – 13:00 Hybrid Intelligence in Practice
13:00 – 14:00 Lunch Break
14:00 – 15:15 Workshop Session 3
15.15 – 15:30 Closing
15:30 – 16:00 Coffee/Tea

29. WHAT TYPES OF PROBLEMS CAN BE SOLVED
THROUGH HYBRID INTELLIGENCE?

30. COMPLEX PROBLEMS
Require Domain Knowledge
Require Empathy or Creativity
Dynamic
No Ground Truth
Require Gut Feeling Stochastic
Time Variant

31. INTUITION
„A nonconscious, holistic processing mode in which judgments are made

32. 2 FORMS OF INTUITION
INTUITIVE
INSIGHT
INTUITIVE
JUDGEMENT

33. BAYES ERROR?
Human
Experienced
Doctor
Group of
Experienced
Doctors
~10% ~5% ~1%
$$$$

34. DESIGNING A HYBRID INTELLIGENCE
SYSTEM THAT HAS A BETTER
INTUITION THAN THE BEST GROUP OF
EXPERTS

35. HYBRID INTELLIGENCE
teaching intuition
teaching intuition

36. INTUITION MACHINE ARCHITECTURE
PREDICTION
EXPERT CROWD
LEARNER
GENERATIVE ENTITY
Reward
Generation Match for Labels
Weigh Labels

37. DESIGN PRINCIPLES
1. Cognitive Feedback on Consistency
2. Social Feedback from Peers
3. Interpretable Machine Feedback
4. Expertise Matching
5. Generative Aggregation
6. Heuristic and Judgement

38. Agenda
Time Top
09:00 – 09:30 Get-together and Introduction
09:30 – 09:45 A short intro to Hybrid Intelligence
09:45 – 10:30 Workshop Session 1a: Paper/Project Presentations
10:30 – 10:50 Coffee Break
11:00 – 12:30 Workshop Session 1b: Paper/Project Presentations
12:30 – 13:00 Hybrid Intelligence in Practice
13:00 – 14:00 Lunch Break
14:00 – 15:15 Workshop Session 3: Workshop on Hybrid Intelligence in IS Research Agenda
15.15 – 15:30 Closing
15:30 – 16:00 Coffee/Tea

39. Platzhalter Vortrag 2
Break – 60min

40. Agenda
Time Top
09:00 – 09:30 Get-together and Introduction
09:30 – 09:45 A short intro to Hybrid Intelligence
09:45 – 10:30 Workshop Session 1a: Paper/Project Presentations
10:30 – 10:50 Coffee Break
11:00 – 12:30 Workshop Session 1b: Paper/Project Presentations
12:30 – 13:00 Hybrid Intelligence in Practice
13:00 – 14:00 Lunch Break
14:00 – 15:15 Workshop Session 3: Workshop on Hybrid Intelligence in IS Research Agenda
15.15 – 15:30 Closing
15:30 – 16:00 Coffee/Tea

41. Towards a Research Agenda
on Hybrid Intelligence

42. Top 1 – Applications
Which technological applications will affect the interaction
between humans and AI in the near future?
Which technological applications will be most relevant for the
design of interactions between humans and AI in the near
future?

43. Top 2 - Research Questions
Which research questions do arise when examining human-
AI-interaction in the identified settings?

44. Top 3 - Research Streams
Is it possible to identify common research streams within the
identified research questions??

45. Agenda
Time Top
09:00 – 09:30 Get-together and Introduction
09:30 – 09:45 A short intro to Hybrid Intelligence
09:45 – 10:30 Workshop Session 1a: Paper/Project Presentations
10:30 – 10:50 Coffee Break
11:00 – 12:30 Workshop Session 1b: Paper/Project Presentations
12:30 – 13:00 Hybrid Intelligence in Practice
13:00 – 14:00 Lunch Break
14:00 – 15:15 Workshop Session 3: Workshop on Hybrid Intelligence in IS Research Agenda
15.15 – 15:30 Closing
15:30 – 16:00 Coffee/Tea

46. Announcement 1
-
WAIM Research Network

47. WAIM Research Network
• WAIM - Work in the Age of Intelligent Machines
• Funded by the US National Science Foundation to coordinate
research on the human-technology frontier (NSF 17–45463)
• Co-PIs Kevin Crowston, Ingrid Erickson and Jeffrey Nickerson
• Goal: Build a network of interested researchers to facilitate
“disciplinary convergence” to address the topic
• Disciplinary convergence (an NSF ”big idea”)
• Research driven by a specific and compelling problem (work and intelligent
machines)
• Deep integration across disciplines by developing common frameworks and
new scientific language

48. WAIM Research Network
• Activities
• Workshops at Academy of Management, HICSS, ECIS and others
• Annual conference (1st held at Michigan State University in May 2018)
• Infrastructure: https://waim.network/, listserv, @WAIMRCN on Twitter
• How we’ll know it’s a success
• The network facilitates joint projects that build convergence
• A network’s strength is its members, so join today!
• https://waim.network/join
• https://waim.network/listserv

49. The Interplay of Machines and Humans:
State of the Art and a Research Agenda for
Hybrid Intelligence