My opening lecture at the »trust in robots« summer school at TU Wien, september 2019 - http://www.tuaustria.ac.at/fileadmin/shares/tuaustria/veranstaltungen/2019-05/Summerschool_Trust_Robots_Programme_final_V1.pdf

1. #fail

2. #fail

3. #fail

4. [...] Die Unfallursache ist voraussichtlich eine menschliche Fehlleistung des zuständigen
Fahrdienstleiters. Nach einer technischen Störung bei einer Weiche (vermutlich
witterungsbedingt) musste das Störungsprozedere vom Fahrdienstleiter in Penzing manuell
in Kraft gesetzt werden, dabei wurde laut vorläufigem Ermittlungsstand vergessen ein Signal
umzustellen.
Railway accident in Vienna:
human error cause for train
collision?

5. Vasa, 1628-1628
1

6. [...] Die Unfallursache ist voraussichtlich eine menschliche Fehlleistung des zuständigen
Fahrdienstleiters. Nach einer technischen Störung bei einer Weiche (vermutlich
witterungsbedingt) musste das Störungsprozedere vom Fahrdienstleiter in Penzing manuell
in Kraft gesetzt werden, dabei wurde laut vorläufigem Ermittlungsstand vergessen ein Signal
umzustellen.
Railway accident in Vienna:
human error cause for train
collision?

7. how can you create technologies
that gracefully recede/degrade
when they don’t work?

8. 2

9. 2

10. Challenger, 1986
2

15. [...] Die Unfallursache ist voraussichtlich eine menschliche Fehlleistung des zuständigen
Fahrdienstleiters. Nach einer technischen Störung bei einer Weiche (vermutlich
witterungsbedingt) musste das Störungsprozedere vom Fahrdienstleiter in Penzing manuell
in Kraft gesetzt werden, dabei wurde laut vorläufigem Ermittlungsstand vergessen ein Signal
umzustellen.
Railway accident in Vienna:
human error cause for train
collision?

16. are we aware of the limitations of
your technology, and can we
communicate them?

17. 3

20. »[…] a touchscreen system that was complex and that
sailors had been poorly trained […]
When we started getting the feedback […] it was really
eye-opening. And it goes into the, in my mind, ›just
because you can doesn’t mean you should‹ category[…]
As a result of innovation and a desire to incorporate new
technology, ›we got away from the physical throttles, and
that was probably the number-one feedback from the
fleet – they said, just give us the throttles that we can
use.‹«
https://news.usni.org/2019/08/09/navy-reverting-ddgs-back-to-physical-throttles-after-fleet-rejects-touchscreen-controls

21. »As the massive B-17 Flying Fortress touched down, the
pilot flicked the control to slow his descent as he had
always done. Within seconds, the massive plane hit the
ground, skidding on its belly, igniting the fuselage and
bursting into flames.«

28. [...] Die Unfallursache ist voraussichtlich eine menschliche Fehlleistung des zuständigen
Fahrdienstleiters. Nach einer technischen Störung bei einer Weiche (vermutlich
witterungsbedingt) musste das Störungsprozedere vom Fahrdienstleiter in Penzing manuell
in Kraft gesetzt werden, dabei wurde laut vorläufigem Ermittlungsstand vergessen ein Signal
umzustellen.
Railway accident in Vienna:
human error cause for train
collision?

29. how can technology contribute to
accidents/disasters or cause other
problems because of how it is
designed/configured/etc.?

30. e-voting
4

31. eVoting - electronic voting machines

32. eVoting - electronic voting machines

33. »Our analysis shows that this voting system is far below
even the most minimal security standards applicable in
other contexts.«
avi rubin, informatiker, john hopkins university
fundstücke u.a.
#defineDESKEY((des_KEY8F2654hd4"

34. »Diebold hadn't password-protected the [vote database]
file or secured the audit log, so anyone with access to the
tabulation program during an election could change votes
and alter the log to erase the evidence.«
kim zetter, wired news

35. eVoting - electronic voting machines

36. eVoting - electronic voting machines

37. eVoting - electronic voting machines

39. »Diebold voting machines opened with hotel minibar key
The key that controls access to a standard Diebold
voting machine is a common key that can be ordered from
the Internet, also used to open hotel minibars.«
boingboing.net

40. »California banned the use of Diebold machines because
the security was so bad.«
Bob Rowe, www.coastalpost.com/04/12/02a.htm

41. »The important thing isn't the voting software, it's an
effective voting procedure.«
comment by chandon seldon on slashdot.org

42. »The important thing isn't the voting software, it's an
effective voting procedure.«
user comment by chandon seldon on slashdot.org

44. Process guide to Austria's E-Voting project:
http://www.oeh-wahl.at
I. Vote Casting between May 18th,
08:00 and May 22nd, 2009, 18:00
1. Encryption with the public key of the
election commission
Identification
Using Citizen Card
May 28th, 2009, 17:00
1. Removal of
the signatur
II. Counting
of the Votes
2. Mixing of the
encrypted ballots
3. Decryption with the private
key of the election commission,
then counting the votes
2. Signing of the completed
encrypted ballot
Hiermit gebe ich,
Max Mustermann,
0800001, die folgenden
verschlüsselten
Stimmen ab.
Hiermit gebe ich,
Max Mustermann,
0800001, die folgenden
verschlüsselten
Stimmen ab.
Hiermit gebe ich,
Max Mustermann,
0800001, die folgenden
verschlüsselten
Stimmen ab.
Hiermit gebe ich,
Max Mustermann,
0800001, die folgenden
verschlüsselten
Stimmen ab.

45. Process guide to Austria's E-Voting project:
http://www.oeh-wahl.at
I. Vote Casting between May 18th,
08:00 and May 22nd, 2009, 18:00
1. Encryption with the public key of the
election commission
Identification
Using Citizen Card
May 28th, 2009, 17:00
1. Removal of
the signatur
II. Counting
of the Votes
2. Mixing of the
encrypted ballots
3. Decryption with the private
key of the election commission,
then counting the votes
2. Signing of the completed
encrypted ballot
Hiermit gebe ich,
Max Mustermann,
0800001, die folgenden
verschlüsselten
Stimmen ab.
Hiermit gebe ich,
Max Mustermann,
0800001, die folgenden
verschlüsselten
Stimmen ab.
Hiermit gebe ich,
Max Mustermann,
0800001, die folgenden
verschlüsselten
Stimmen ab.
Hiermit gebe ich,
Max Mustermann,
0800001, die folgenden
verschlüsselten
Stimmen ab.
➔
wahlkommission
wähler/innen
trust? black box

46. ursachen
systemische
bedingungen
trust?

47. ursachen
systemische
bedingungen
»TNO«-security
steve gibson
(trust noone)

49. implementation and use of
technologies transforms
mathematical concepts that are
provably correct into techno-social
»alloys«

50. malware in personal computers
Report: 48% of 22 million scanned computers infected
with malware [ZDnet]
32% of computers around the world are infected with
viruses and malware [dotTech]
Malware infects 30% of computers in the U.S.
[InfoWorld]

51. Technology is neither
good nor bad —
nor is it neutral.

52. making
technology
has an ethical
dimension
that cannot
be silenced by
ignoring it.

53. who will benefit from the
technology we create? who will
lose?
what is now common, but will
then become hard or impossible?

54. [...] Die Unfallursache ist voraussichtlich eine menschliche Fehlleistung des zuständigen
Fahrdienstleiters. Nach einer technischen Störung bei einer Weiche (vermutlich
witterungsbedingt) musste das Störungsprozedere vom Fahrdienstleiter in Penzing manuell
in Kraft gesetzt werden, dabei wurde laut vorläufigem Ermittlungsstand vergessen ein Signal
umzustellen.
Railway accident in Vienna:
human error cause for train
collision?

55. rail dispatcher
5

56. five sources of danger in automation
risk compensation

57. five sources of danger in automation
risk compensation
irony of automisation

59. Many Aviators Have Difficulty
Manually Flying Planes, Study
Commissioned by FAA Finds
»Commercial airline pilots have
become so dependent on
automation that poor manual
flying skills and failure to master
the latest changes in cockpit
technology pose the greatest
hazards to passengers, an
international panel of air-safety
experts warns.«

60. five sources of danger in automation
risk compensation
irony of automisation
stress

61. five sources of danger in automation
risk compensation
irony of automisation
stress
errors

62. five sources of danger in automation
risk compensation
irony of automisation
stress
errors
the »bathtub curve«

64. throughput
efficiency
automation
complexity

65. throughput
efficiency
automation
complexity
complexity

66. [...] Die Unfallursache ist voraussichtlich eine menschliche Fehlleistung des zuständigen
Fahrdienstleiters. Nach einer technischen Störung bei einer Weiche (vermutlich
witterungsbedingt) musste das Störungsprozedere vom Fahrdienstleiter in Penzing manuell
in Kraft gesetzt werden, dabei wurde laut vorläufigem Ermittlungsstand vergessen ein Signal
umzustellen.
Railway accident in Vienna:
human error cause for train
collision?

67. absurd consequence: automation
justifies more automation

68. Cheaper, better robots will replace human workers in the
world's factories at a faster pace over the next decade,
pushing labor costs down 16%, a report Tuesday said[…]
Robots will cut labor costs by 33% in South Korea, 25%
in Japan, 24% in Canada and 22% in the United States
and Taiwan.
The cost of owning and operating a robotic spot welder,
for instance, has tumbled from $182,000 in 2005 to
$133,000 last year, and will drop to $103,000 by 2025

71. »›There are approximately 4 million truck, taxi, limousine
and bus drivers in the United States, not to mention gas
station attendants and traffic policemen,‹ writes Posner,
the University of Chicago scholar, in his essay on
automation and employment. ›Not all these jobs will be
eliminated overnight,‹ he says, ›but they could go quite
fast.‹«
Der Spiegel

72. automation is rarely designed
to regard the interests of
more than a few people.

73. »The progress in AI research
makes it timely to focus research
not only on making AI more
capable, but also on maximizing
the societal benefit of AI.«

74. how can we design technologies to
be human-centered instead of
»just because we can«?

75. 6
self-driving cars

76. »Human drivers may be forgiven for making an instinctive
but nonetheless bad split-second decision, such as
swerving into incoming traffic rather than the other way
into a field. But programmers and designers of automated
cars don’t have that luxury, since they do have the time to
get it right and therefore bear more responsibility for bad
outcomes.«
Lin, P. 2013, The Ethics of Autonomous Cars,
The Atlantic.

77. »[…] an incident over the weekend in which the owner
of a Tesla Model X SUV crashed into a building and claimed
it had suddenly accelerated on its own. But Tesla vehicles
are constantly connected to their manufacturer via the
Internet, and the company had this to say in a
statement to the Verge:
›Data shows that the vehicle was traveling at 6 mph
when the accelerator pedal was abruptly increased to 100
percent.‹«
Tesla Knows When a Crash Is Your Fault, and Other
Carmakers Soon Will, Too. MIT Technology Review

81. »Legislation – preferably world-wide on a similar basis to civil
aviation – is needed now.
Product liability should be relaxed for driverless cars.
Accidents should be investigated on a no-blame model and
recommendations made to avoid any repetition.
All commercial algorithms for this sort of control decision should
be made public.
All design and development of driverless software should be
legally required to have open ethical scrutiny at all stages.«
Blay Whitby (2016), technology ethicist

82. »The progress in AI research
makes it timely to focus research
not only on making AI more
capable, but also on maximizing
the societal benefit of AI.«

83. 1. Instead of preventing misuse: reduce effects, minimise damages
2. Instead of automation and take-over: technology as support, human
cooperation as basis
3. Instead of monopolies: keep alternatives, secure plurality
4. Instead of reliance on a single solution: create redundancies
5. Instead of dependence on a single vendor: temporal, spacial,
technological and organisational diversity
6. Instead of centralisation: decoupled, transparent, decentral solutions
7. Instead of total failure: reach stable state in partial or complete failure
8. Instead of surprises: systematic planning for emergencies
9. Instead of heteronomy: design of technology, esp. safety systems, with
approval of concerned parties and the public
10. Renounce economic advantage if potential damage is too high
alexander roßnagel:
10 design-proposals
1989
roßnagel et al, die verletzlichkeit
der »informationsgesellschaft«

88. how can we design
technologies to be
human-centered
instead of »just
because we can«?
who will benefit from
the technology we
create? who will lose?
what is now common,
but will then become
hard or impossible?
how can technology
contribute to
accidents/disasters or
cause other problems
because of how it is
designed/configured/
etc.?
how can technology
contribute to
accidents/disasters or
cause other problems
because of how it is
designed/configured/
etc.?
how can you create
technologies that
gracefully recede/
degrade when they
don’t work?

90. »Ich war nur Techniker_In«

91. schrift
yanone kaffeesatz, (cc) jan gerner
ENDE